Grokbase › Groups › Node.js › nodejs › August 2012
FAQ
Hi Everyone,

I am fairly new to Node.js and I have a hard time trying to figure this one
out. Here's I am trying to do:

I have a node instance acting as a master program who will spawn other long
running applications (possible up to 100 programs) (that is not a node.js
instance) as worker applications. I would like my node instance to be able
to communicate with them via a REPL style, that is 1) node would issue
"work commands" to the worker applications, 2) and the worker applications
would be reply with "acknowledgements", "work done" or "error" back to the
node, so 3) node.js could effective schedule jobs and optimize the worker
applications.

The problem here is that the streams between node and the child_process are
buffered by default to a length of 8192 bytes, which means that I have to
pad my message until it is of this length which is inelegant. I want a much
smaller buffer for message, which presumably would be give less
communications overhead, and I dont want to be counting characters to make
just they return in the same buffer at every message response. Making
stream emit a "data" on detection of a "\0" character would be idea.

I have tried, various custom stdio options for child_process<http://nodejs.org/api/child_process.html#child_process_child_process_spawn_command_args_options>,
to try to modify the buffer length but it was to no avail. Help or pointers
would be very much appreciated!

Cheers
Tomithy



--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
nodejs+[email protected]
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

Search Discussions

  • Tim Caswell at Aug 21, 2012 at 1:20 pm ⇧
    I implement my own protocols over stdio all the time. Node doesn't wait
    till 8k of data is queued before sending. You just need to treat the same
    as a TCP stream and not assume anything about the chunk sizes. The
    simplest framing protocol for binary data is to send a 4byte length header
    before each message. Then on the receiving end, implement an
    interruptable state machine parser. Have this parser emit message events
    with the original message bodies reconstructed and do you
    protocol/application logic from there.
    On Aug 21, 2012 5:31 AM, "Tomithy Too" wrote:

    Hi Everyone,

    I am fairly new to Node.js and I have a hard time trying to figure this
    one out. Here's I am trying to do:

    I have a node instance acting as a master program who will spawn other
    long running applications (possible up to 100 programs) (that is not a
    node.js instance) as worker applications. I would like my node instance to
    be able to communicate with them via a REPL style, that is 1) node would
    issue "work commands" to the worker applications, 2) and the worker
    applications would be reply with "acknowledgements", "work done" or "error"
    back to the node, so 3) node.js could effective schedule jobs and optimize
    the worker applications.

    The problem here is that the streams between node and the child_process
    are buffered by default to a length of 8192 bytes, which means that I have
    to pad my message until it is of this length which is inelegant. I want a
    much smaller buffer for message, which presumably would be give less
    communications overhead, and I dont want to be counting characters to make
    just they return in the same buffer at every message response. Making
    stream emit a "data" on detection of a "\0" character would be idea.

    I have tried, various custom stdio options for child_process<http://nodejs.org/api/child_process.html#child_process_child_process_spawn_command_args_options>,
    to try to modify the buffer length but it was to no avail. Help or pointers
    would be very much appreciated!

    Cheers
    Tomithy



    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines:
    https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to [email protected]
    To unsubscribe from this group, send email to
    nodejs+[email protected]
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to [email protected]
    To unsubscribe from this group, send email to
    nodejs+[email protected]
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
  • Tomithy Too at Aug 22, 2012 at 2:48 pm ⇧
    Hi Tim,

    Thanks for your reply. I am wondering if you could provide me with some
    code example of it.

    I would like to clarify that for my instance, the child process are running
    in the same server as the master node.js process, ie the messages are not
    transferred over a network, and if possible i prefer that it does not go
    through the network stack.

    Cheers
    Tomithy

    On Tuesday, August 21, 2012 9:20:31 PM UTC+8, Tim Caswell wrote:

    I implement my own protocols over stdio all the time. Node doesn't wait
    till 8k of data is queued before sending. You just need to treat the same
    as a TCP stream and not assume anything about the chunk sizes. The
    simplest framing protocol for binary data is to send a 4byte length header
    before each message. Then on the receiving end, implement an
    interruptable state machine parser. Have this parser emit message events
    with the original message bodies reconstructed and do you
    protocol/application logic from there.
    On Aug 21, 2012 5:31 AM, "Tomithy Too" <[email protected] <javascript:>>
    wrote:
    Hi Everyone,

    I am fairly new to Node.js and I have a hard time trying to figure this
    one out. Here's I am trying to do:

    I have a node instance acting as a master program who will spawn other
    long running applications (possible up to 100 programs) (that is not a
    node.js instance) as worker applications. I would like my node instance to
    be able to communicate with them via a REPL style, that is 1) node would
    issue "work commands" to the worker applications, 2) and the worker
    applications would be reply with "acknowledgements", "work done" or "error"
    back to the node, so 3) node.js could effective schedule jobs and optimize
    the worker applications.

    The problem here is that the streams between node and the child_process
    are buffered by default to a length of 8192 bytes, which means that I have
    to pad my message until it is of this length which is inelegant. I want a
    much smaller buffer for message, which presumably would be give less
    communications overhead, and I dont want to be counting characters to make
    just they return in the same buffer at every message response. Making
    stream emit a "data" on detection of a "\0" character would be idea.

    I have tried, various custom stdio options for child_process<http://nodejs.org/api/child_process.html#child_process_child_process_spawn_command_args_options>,
    to try to modify the buffer length but it was to no avail. Help or pointers
    would be very much appreciated!

    Cheers
    Tomithy



    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines:
    https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to [email protected]<javascript:>
    To unsubscribe from this group, send email to
    [email protected] <javascript:>
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to [email protected]
    To unsubscribe from this group, send email to
    nodejs+[email protected]
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
  • Tim Caswell at Aug 22, 2012 at 3:18 pm ⇧
    Ok, the deframer I use in smith (a node rpc protocol I designed) is at
    https://github.com/c9/smith/blob/master/smith.js#L150-191

    It's a function that takes an onMessage callback and returns a
    function that consumes bytes.
    https://github.com/c9/smith/blob/master/smith.js#L85-94

    To encode frames on the sending side, no state machine is required.
    Simply send the header and the body.
    https://github.com/c9/smith/blob/master/smith.js#L137-147

    I use this over local stdio as well as over tcp sockets. It doesn't
    care at all that the chunk size ends up being. I unit test it by
    sending chunks from every size between 1 byte per chunk and the whole
    message in a single chunk.
    https://github.com/c9/smith/blob/master/tests/test-framer.js
    On Wed, Aug 22, 2012 at 9:47 AM, Tomithy Too wrote:
    Hi Tim,

    Thanks for your reply. I am wondering if you could provide me with some code
    example of it.

    I would like to clarify that for my instance, the child process are running
    in the same server as the master node.js process, ie the messages are not
    transferred over a network, and if possible i prefer that it does not go
    through the network stack.

    Cheers
    Tomithy


    On Tuesday, August 21, 2012 9:20:31 PM UTC+8, Tim Caswell wrote:

    I implement my own protocols over stdio all the time. Node doesn't wait
    till 8k of data is queued before sending. You just need to treat the same
    as a TCP stream and not assume anything about the chunk sizes. The simplest
    framing protocol for binary data is to send a 4byte length header before
    each message. Then on the receiving end, implement an interruptable state
    machine parser. Have this parser emit message events with the original
    message bodies reconstructed and do you protocol/application logic from
    there.
    On Aug 21, 2012 5:31 AM, "Tomithy Too" wrote:

    Hi Everyone,

    I am fairly new to Node.js and I have a hard time trying to figure this
    one out. Here's I am trying to do:

    I have a node instance acting as a master program who will spawn other
    long running applications (possible up to 100 programs) (that is not a
    node.js instance) as worker applications. I would like my node instance to
    be able to communicate with them via a REPL style, that is 1) node would
    issue "work commands" to the worker applications, 2) and the worker
    applications would be reply with "acknowledgements", "work done" or "error"
    back to the node, so 3) node.js could effective schedule jobs and optimize
    the worker applications.

    The problem here is that the streams between node and the child_process
    are buffered by default to a length of 8192 bytes, which means that I have
    to pad my message until it is of this length which is inelegant. I want a
    much smaller buffer for message, which presumably would be give less
    communications overhead, and I dont want to be counting characters to make
    just they return in the same buffer at every message response. Making stream
    emit a "data" on detection of a "\0" character would be idea.

    I have tried, various custom stdio options for child_process, to try to
    modify the buffer length but it was to no avail. Help or pointers would be
    very much appreciated!

    Cheers
    Tomithy



    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines:
    https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to [email protected]

    To unsubscribe from this group, send email to
    [email protected]

    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines:
    https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to [email protected]
    To unsubscribe from this group, send email to
    nodejs+[email protected]
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to [email protected]
    To unsubscribe from this group, send email to
    nodejs+[email protected]
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
  • Mark Hahn at Aug 22, 2012 at 6:32 pm ⇧
    if possible i prefer that it does not go through the network stack.
    In linux, the network stack is smart and recognizes when two endpoints are
    on the same machine. In that case it short-circuits the stack and treats
    the data much the same as a unix pipe. So you never have to worry about
    going through the stack because you won't.
    On Wed, Aug 22, 2012 at 8:18 AM, Tim Caswell wrote:

    Ok, the deframer I use in smith (a node rpc protocol I designed) is at
    https://github.com/c9/smith/blob/master/smith.js#L150-191

    It's a function that takes an onMessage callback and returns a
    function that consumes bytes.
    https://github.com/c9/smith/blob/master/smith.js#L85-94

    To encode frames on the sending side, no state machine is required.
    Simply send the header and the body.
    https://github.com/c9/smith/blob/master/smith.js#L137-147

    I use this over local stdio as well as over tcp sockets. It doesn't
    care at all that the chunk size ends up being. I unit test it by
    sending chunks from every size between 1 byte per chunk and the whole
    message in a single chunk.
    https://github.com/c9/smith/blob/master/tests/test-framer.js
    On Wed, Aug 22, 2012 at 9:47 AM, Tomithy Too wrote:
    Hi Tim,

    Thanks for your reply. I am wondering if you could provide me with some code
    example of it.

    I would like to clarify that for my instance, the child process are running
    in the same server as the master node.js process, ie the messages are not
    transferred over a network, and if possible i prefer that it does not go
    through the network stack.

    Cheers
    Tomithy


    On Tuesday, August 21, 2012 9:20:31 PM UTC+8, Tim Caswell wrote:

    I implement my own protocols over stdio all the time. Node doesn't wait
    till 8k of data is queued before sending. You just need to treat the
    same
    as a TCP stream and not assume anything about the chunk sizes. The
    simplest
    framing protocol for binary data is to send a 4byte length header before
    each message. Then on the receiving end, implement an interruptable
    state
    machine parser. Have this parser emit message events with the original
    message bodies reconstructed and do you protocol/application logic from
    there.
    On Aug 21, 2012 5:31 AM, "Tomithy Too" wrote:

    Hi Everyone,

    I am fairly new to Node.js and I have a hard time trying to figure this
    one out. Here's I am trying to do:

    I have a node instance acting as a master program who will spawn other
    long running applications (possible up to 100 programs) (that is not a
    node.js instance) as worker applications. I would like my node
    instance to
    be able to communicate with them via a REPL style, that is 1) node
    would
    issue "work commands" to the worker applications, 2) and the worker
    applications would be reply with "acknowledgements", "work done" or
    "error"
    back to the node, so 3) node.js could effective schedule jobs and
    optimize
    the worker applications.

    The problem here is that the streams between node and the child_process
    are buffered by default to a length of 8192 bytes, which means that I
    have
    to pad my message until it is of this length which is inelegant. I
    want a
    much smaller buffer for message, which presumably would be give less
    communications overhead, and I dont want to be counting characters to
    make
    just they return in the same buffer at every message response. Making
    stream
    emit a "data" on detection of a "\0" character would be idea.

    I have tried, various custom stdio options for child_process, to try to
    modify the buffer length but it was to no avail. Help or pointers
    would be
    very much appreciated!

    Cheers
    Tomithy



    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines:
    https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to [email protected]

    To unsubscribe from this group, send email to
    [email protected]

    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines:
    https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to [email protected]
    To unsubscribe from this group, send email to
    nodejs+[email protected]
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines:
    https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to [email protected]
    To unsubscribe from this group, send email to
    nodejs+[email protected]
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to [email protected]
    To unsubscribe from this group, send email to
    nodejs+[email protected]
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupnodejs @
categoriesnodejs
postedAug 21, '12 at 10:31a
activeAug 22, '12 at 6:32p
posts5
users3
websitenodejs.org
irc#node.js

3 users in discussion

Tim Caswell: 2 posts Tomithy Too: 2 posts Mark Hahn: 1 post

Content

People

Support

Translate

site design / logo © 2023 Grokbase