FAQ
I'm writing a node.js server that uses the cluster module. Each worker
does *lots* of logging to a shared log file (it must be just 1 file).

- Assuming log lines are smaller than PIPE_BUF (4k), is it safe to do
fs.write() and rely on it being atomic, or does Node.js break this
promise of write()?

- What's with "Note that it is unsafe to use fs.write multiple times
on the same file without waiting for the callback."?

- If writes are not atomic or not waiting for callbacks really is a
problem, what's an *efficient* alternative?

Thanks,
Jaka

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

Search Discussions

  • Mark Hahn at Mar 24, 2012 at 6:43 pm
    Writes are atomic just like calling them from any other language.

    The warning is just pointing out that due to the asynchronous nature of
    javascript you must not assume any IO has happened until you get the
    callback. So if you don't wait for callbacks the order of the writes is
    not guaranteed. In your case of writing to a file from different places in
    your app, you wouldn't expect the order of writes to be guaranteed anyway.
    On Sat, Mar 24, 2012 at 10:08 AM, Jaka Jančar wrote:

    I'm writing a node.js server that uses the cluster module. Each worker
    does *lots* of logging to a shared log file (it must be just 1 file).

    - Assuming log lines are smaller than PIPE_BUF (4k), is it safe to do
    fs.write() and rely on it being atomic, or does Node.js break this
    promise of write()?

    - What's with "Note that it is unsafe to use fs.write multiple times
    on the same file without waiting for the callback."?

    - If writes are not atomic or not waiting for callbacks really is a
    problem, what's an *efficient* alternative?

    Thanks,
    Jaka

    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines:
    https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
  • Jaka Jančar at Mar 24, 2012 at 9:38 pm
    I was thinking more along the lines:

    If I do:

    fs.write(fd, 'first write of two kilobytes');
    fs.write(fd, 'second write three kilobytes');

    is there a userspace buffer involved somewhere that could cause e.g. 4
    kilobytes to be written (2k of first and 2k of second write), then write
    the remaining kilobyte sometime later, thus manging the record?
    On Saturday, March 24, 2012 7:42:35 PM UTC+1, Mark Hahn wrote:

    Writes are atomic just like calling them from any other language.

    The warning is just pointing out that due to the asynchronous nature of
    javascript you must not assume any IO has happened until you get the
    callback. So if you don't wait for callbacks the order of the writes is
    not guaranteed. In your case of writing to a file from different places in
    your app, you wouldn't expect the order of writes to be guaranteed anyway.
    On Sat, Mar 24, 2012 at 10:08 AM, Jaka Jančar wrote:

    I'm writing a node.js server that uses the cluster module. Each worker
    does *lots* of logging to a shared log file (it must be just 1 file).

    - Assuming log lines are smaller than PIPE_BUF (4k), is it safe to do
    fs.write() and rely on it being atomic, or does Node.js break this
    promise of write()?

    - What's with "Note that it is unsafe to use fs.write multiple times
    on the same file without waiting for the callback."?

    - If writes are not atomic or not waiting for callbacks really is a
    problem, what's an *efficient* alternative?

    Thanks,
    Jaka

    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines:
    https://github.com/joyent/​node/wiki/Mailing-List-​Posting-Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines>
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@​googlegroups.com<nodejs%2Bunsubscribe@googlegroups.com>
    For more options, visit this group at
    http://groups.google.com/​group/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en>
    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
  • Jorge at Mar 24, 2012 at 10:05 pm

    On Mar 24, 2012, at 6:08 PM, Jaka Jančar wrote:

    I'm writing a node.js server that uses the cluster module. Each worker
    does *lots* of logging to a shared log file (it must be just 1 file).

    - Assuming log lines are smaller than PIPE_BUF (4k), is it safe to do
    fs.write() and rely on it being atomic, or does Node.js break this
    promise of write()?

    - What's with "Note that it is unsafe to use fs.write multiple times
    on the same file without waiting for the callback."?

    - If writes are not atomic or not waiting for callbacks really is a
    problem, what's an *efficient* alternative?
    When you have several fs.write()s going on at once, they may end up being executed in parallel in a bunch of (up to 4 IIRC) background (libeio's) threads, so the write()s may actually happen in ~ any order. If you instead wait for the callback before issuing the next write(), then you're serializing them.
    --
    Jorge.

    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
  • Jaka Jančar at Mar 24, 2012 at 10:19 pm
    That is besides the point. The question is, will the fs.write()'s map 1:1
    to write()'s?
    On Saturday, March 24, 2012 11:05:25 PM UTC+1, Jorge wrote:
    On Mar 24, 2012, at 6:08 PM, Jaka Jančar wrote:

    I'm writing a node.js server that uses the cluster module. Each worker
    does *lots* of logging to a shared log file (it must be just 1 file).

    - Assuming log lines are smaller than PIPE_BUF (4k), is it safe to do
    fs.write() and rely on it being atomic, or does Node.js break this
    promise of write()?

    - What's with "Note that it is unsafe to use fs.write multiple times
    on the same file without waiting for the callback."?

    - If writes are not atomic or not waiting for callbacks really is a
    problem, what's an *efficient* alternative?
    When you have several fs.write()s going on at once, they may end up being
    executed in parallel in a bunch of (up to 4 IIRC) background (libeio's)
    threads, so the write()s may actually happen in ~ any order. If you instead
    wait for the callback before issuing the next write(), then you're
    serializing them.
    --
    Jorge.
    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
  • Mark Hahn at Mar 24, 2012 at 10:31 pm
    If you are doing append writes it is always atomic.
    On Sat, Mar 24, 2012 at 3:19 PM, Jaka Jančar wrote:

    That is besides the point. The question is, will the fs.write()'s map 1:1
    to write()'s?

    On Saturday, March 24, 2012 11:05:25 PM UTC+1, Jorge wrote:
    On Mar 24, 2012, at 6:08 PM, Jaka Jančar wrote:

    I'm writing a node.js server that uses the cluster module. Each worker
    does *lots* of logging to a shared log file (it must be just 1 file).

    - Assuming log lines are smaller than PIPE_BUF (4k), is it safe to do
    fs.write() and rely on it being atomic, or does Node.js break this
    promise of write()?

    - What's with "Note that it is unsafe to use fs.write multiple times
    on the same file without waiting for the callback."?

    - If writes are not atomic or not waiting for callbacks really is a
    problem, what's an *efficient* alternative?
    When you have several fs.write()s going on at once, they may end up being
    executed in parallel in a bunch of (up to 4 IIRC) background (libeio's)
    threads, so the write()s may actually happen in ~ any order. If you instead
    wait for the callback before issuing the next write(), then you're
    serializing them.
    --
    Jorge.

    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines:
    https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
  • Jorge at Mar 24, 2012 at 10:39 pm
    The answer is *no*.
    On Mar 24, 2012, at 11:19 PM, Jaka Jančar wrote:

    That is besides the point. The question is, will the fs.write()'s map 1:1 to write()'s?

    On Saturday, March 24, 2012 11:05:25 PM UTC+1, Jorge wrote:
    On Mar 24, 2012, at 6:08 PM, Jaka Jančar wrote:
    I'm writing a node.js server that uses the cluster module. Each worker
    does *lots* of logging to a shared log file (it must be just 1 file).

    - Assuming log lines are smaller than PIPE_BUF (4k), is it safe to do
    fs.write() and rely on it being atomic, or does Node.js break this
    promise of write()?

    - What's with "Note that it is unsafe to use fs.write multiple times
    on the same file without waiting for the callback."?

    - If writes are not atomic or not waiting for callbacks really is a
    problem, what's an *efficient* alternative?
    When you have several fs.write()s going on at once, they may end up being executed in parallel in a bunch of (up to 4 IIRC) background (libeio's) threads, so the write()s may actually happen in ~ any order. If you instead wait for the callback before issuing the next write(), then you're serializing them.
    --
    Jorge.



    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
  • Jorge at Mar 24, 2012 at 10:45 pm
    And WRT "That is besides the point": you've asked "What's with "Note that it is unsafe to use fs.write multiple times on the same file without waiting for the callback.?"
    On Mar 24, 2012, at 11:19 PM, Jaka Jančar wrote:

    That is besides the point. The question is, will the fs.write()'s map 1:1 to write()'s?

    On Saturday, March 24, 2012 11:05:25 PM UTC+1, Jorge wrote:
    On Mar 24, 2012, at 6:08 PM, Jaka Jančar wrote:
    I'm writing a node.js server that uses the cluster module. Each worker
    does *lots* of logging to a shared log file (it must be just 1 file).

    - Assuming log lines are smaller than PIPE_BUF (4k), is it safe to do
    fs.write() and rely on it being atomic, or does Node.js break this
    promise of write()?

    - What's with "Note that it is unsafe to use fs.write multiple times
    on the same file without waiting for the callback."?

    - If writes are not atomic or not waiting for callbacks really is a
    problem, what's an *efficient* alternative?
    When you have several fs.write()s going on at once, they may end up being executed in parallel in a bunch of (up to 4 IIRC) background (libeio's) threads, so the write()s may actually happen in ~ any order. If you instead wait for the callback before issuing the next write(), then you're serializing them.
    --
    Jorge.



    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
  • Micheil Smith at Mar 26, 2012 at 10:05 am
    I think you may actually be able to use a WriteStream instead, as far as I recall,
    those queued data up in an array, then they write it out as soon as possible,
    which should guarantee ordering.

    Alternatively, since this is a log file, you could use something like the winston
    by Charlie Robbins.

    – Micheil
    On 24/03/2012, at 10:45 PM, Jorge wrote:

    And WRT "That is besides the point": you've asked "What's with "Note that it is unsafe to use fs.write multiple times on the same file without waiting for the callback.?"
    On Mar 24, 2012, at 11:19 PM, Jaka Jančar wrote:

    That is besides the point. The question is, will the fs.write()'s map 1:1 to write()'s?

    On Saturday, March 24, 2012 11:05:25 PM UTC+1, Jorge wrote:
    On Mar 24, 2012, at 6:08 PM, Jaka Jančar wrote:
    I'm writing a node.js server that uses the cluster module. Each worker
    does *lots* of logging to a shared log file (it must be just 1 file).

    - Assuming log lines are smaller than PIPE_BUF (4k), is it safe to do
    fs.write() and rely on it being atomic, or does Node.js break this
    promise of write()?

    - What's with "Note that it is unsafe to use fs.write multiple times
    on the same file without waiting for the callback."?

    - If writes are not atomic or not waiting for callbacks really is a
    problem, what's an *efficient* alternative?
    When you have several fs.write()s going on at once, they may end up being executed in parallel in a bunch of (up to 4 IIRC) background (libeio's) threads, so the write()s may actually happen in ~ any order. If you instead wait for the callback before issuing the next write(), then you're serializing them.
    --
    Jorge.



    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupnodejs @
categoriesnodejs
postedMar 24, '12 at 5:08p
activeMar 26, '12 at 10:05a
posts9
users4
websitenodejs.org
irc#node.js

People

Translate

site design / logo © 2022 Grokbase