FAQ
Doing some tests on windows with node v0.8.11
--------------------------------------------
var buf = new Buffer(10000000);
buf.fill(0xFF);

var len = buf.length;
var a;
var start = new Date();
for(var b = 0 ; b < len; b++){
a = buf.readUInt8(b);
}
var end = new Date();

console.log('Buffer reads ' + (end - start) + ' ms'); // Prints 81ms

------------------------------------------------

var arr = [];
var len = 10000000;
for(b = 0 ; b < len; b++){
arr[b] = 0xFF;
}
var len = arr.length;
console.log(len);
start = new Date();
for(b = 0 ; b < len; b++){
a = arr[b];
}
end = new Date();

console.log('Array reads ' + (end - start) + ' ms'); // Prints 29ms

And I always thought that buffers would be faster than arrays in node.js ?
:(
Am I doing something wrong ??



--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nodejs@googlegroups.com
To unsubscribe from this group, send email to
nodejs+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

Search Discussions

  • Ben Noordhuis at Oct 3, 2012 at 11:43 pm

    On Thu, Oct 4, 2012 at 1:37 AM, NodeNinja wrote:
    Doing some tests on windows with node v0.8.11
    --------------------------------------------
    var buf = new Buffer(10000000);
    buf.fill(0xFF);

    var len = buf.length;
    var a;
    var start = new Date();
    for(var b = 0 ; b < len; b++){
    a = buf.readUInt8(b);
    }
    var end = new Date();

    console.log('Buffer reads ' + (end - start) + ' ms'); // Prints 81ms

    ------------------------------------------------

    var arr = [];
    var len = 10000000;
    for(b = 0 ; b < len; b++){
    arr[b] = 0xFF;
    }
    var len = arr.length;
    console.log(len);
    start = new Date();
    for(b = 0 ; b < len; b++){
    a = arr[b];
    }
    end = new Date();

    console.log('Array reads ' + (end - start) + ' ms'); // Prints 29ms

    And I always thought that buffers would be faster than arrays in node.js ?
    :(
    Am I doing something wrong ??
    Don't use buf.readUInt8(), it's only there for the sake of parity with
    the other .readUInt*() functions. Replace it with `a = buf[b]` and
    you'll see a 4-5x speed-up.

    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
  • NodeNinja at Oct 3, 2012 at 11:51 pm

    On Thursday, October 4, 2012 5:13:30 AM UTC+5:30, Ben Noordhuis wrote:

    Don't use buf.readUInt8(), it's only there for the sake of parity with
    the other .readUInt*() functions. Replace it with `a = buf[b]` and
    you'll see a 4-5x speed-up.
    Excellent suggestion Ben:
    New results are exciting :)

    Buffer reads 21 ms
    Array reads 29 ms

    What about the other functions like readUIntXX and readIntXX ,I may not be
    able to use indexing in those cases.
    Or will they be as fast as using indexing?


    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
  • Ben Noordhuis at Oct 4, 2012 at 12:14 am

    On Thu, Oct 4, 2012 at 1:51 AM, NodeNinja wrote:
    On Thursday, October 4, 2012 5:13:30 AM UTC+5:30, Ben Noordhuis wrote:


    Don't use buf.readUInt8(), it's only there for the sake of parity with
    the other .readUInt*() functions. Replace it with `a = buf[b]` and
    you'll see a 4-5x speed-up.

    Excellent suggestion Ben:
    New results are exciting :)

    Buffer reads 21 ms
    Array reads 29 ms

    What about the other functions like readUIntXX and readIntXX ,I may not be
    able to use indexing in those cases.
    Or will they be as fast as using indexing?
    They won't, they're there mostly for convenience*. If you need to read
    in e.g. a little-endian short, do this:

    v = buf[i] | (buf[i + 1] << 8);

    Same but big-endian:

    v = (buf[i] << 8) | buf[i + 1];

    * Except for the float and double functions - reading and writing IEEE
    754 floating point numbers is *hard*.

    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
  • NodeNinja at Oct 4, 2012 at 12:38 am

    On Thursday, October 4, 2012 5:44:16 AM UTC+5:30, Ben Noordhuis wrote:
    They won't, they're there mostly for convenience*.
    Now that's an* eye opener* Ben!
    I hope that line finds its way to the documentation.
    Many thanks!!



    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
  • Jimb Esser at Oct 4, 2012 at 4:19 pm
    If you want *really* fast buffer access for larger integer types and float
    types, you can set up a native module that
    calls SetIndexedPropertiesToExternalArrayData a few times on your Buffer
    and gives you a native indexed view into the buffer for whatever primitive
    type you want to read (much, much, much faster than trying to read a float
    byte-by-byte and assemble it). We do this in our internal networking
    buffers, and got quite a speed-up compared to combining bytes in JS (though
    either are *way* faster than calling into native methods to do the read,
    except maybe with floats). Note: for un-aligned reads (e.g. reading a
    64-bit float from 3 bytes within a buffer), I think this only works on some
    platforms, but those platforms include x86 and x64, which is all we care
    about in our case.

    Jimb Esser
    Cloud Party
    On Wednesday, October 3, 2012 5:38:38 PM UTC-7, NodeNinja wrote:


    On Thursday, October 4, 2012 5:44:16 AM UTC+5:30, Ben Noordhuis wrote:

    They won't, they're there mostly for convenience*.
    Now that's an* eye opener* Ben!
    I hope that line finds its way to the documentation.
    Many thanks!!


    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
  • NodeNinja at Oct 4, 2012 at 11:11 pm

    On Thursday, October 4, 2012 9:49:20 PM UTC+5:30, Jimb Esser wrote:
    If you want *really* fast buffer access for larger integer types and float
    types, you can set up a native module that
    calls SetIndexedPropertiesToExternalArrayData a few times on your Buffer
    and gives you a native indexed view into the buffer for whatever primitive
    type you want to read.

    Is there any existing module that does this?
    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
  • Ben Noordhuis at Oct 5, 2012 at 2:27 am

    On Thu, Oct 4, 2012 at 6:19 PM, Jimb Esser wrote:
    If you want *really* fast buffer access for larger integer types and float
    types, you can set up a native module that calls
    SetIndexedPropertiesToExternalArrayData a few times on your Buffer and gives
    you a native indexed view into the buffer for whatever primitive type you
    want to read (much, much, much faster than trying to read a float
    byte-by-byte and assemble it). We do this in our internal networking
    buffers, and got quite a speed-up compared to combining bytes in JS (though
    either are *way* faster than calling into native methods to do the read,
    except maybe with floats). Note: for un-aligned reads (e.g. reading a
    64-bit float from 3 bytes within a buffer), I think this only works on some
    platforms, but those platforms include x86 and x64, which is all we care
    about in our case.
    You don't have to drop down to C++ for that, node.js supports typed
    arrays[1]. I probably should have mentioned that in my other post. :-)

    [1] https://developer.mozilla.org/en-US/docs/JavaScript_typed_arrays

    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
  • Jimb Esser at Oct 5, 2012 at 6:59 am
    Ah, yeah, it does now! We wrote our module back on node 0.4, which, I
    think, did not have typed arrays. That being said, it seems TypedArrays do
    not support un-aligned views (e.g. if you're reading a stream and want to
    read a F64 at byte offset 3, it throws an exception), as well they
    shouldn't, as that's not guaranteed to be efficient cross-platform, though
    is critical to performance in our specific situation. Also, it doesn't
    seem you can make a typed array view off of a Buffer (or get an ArrayBuffer
    off of a network read?), so that involves some copying, although since
    creating views is a native call and a bit expensive, we found it quickest
    to copy any network-received Buffer into one that already has views created
    anyway.

    Sorry, no public module for this, ours is pretty deeply entangled with a
    number of our other systems, and involves careful
    allocating/releasing/pooling of these buffers to be most efficient. The
    gist of it is just to take a Buffer, for each type you want to read (say,
    e.g, 32-bit ints), make a number of views equal to the possible byte
    offsets (e.g. 4) and then do a read with the right offset from the right
    view (e.g. view[offset % 4][offset >> 2]). V8 does "indexed properties to
    external data" lookups super-fast (Buffers use this), so this ends up being
    much faster than doing the JS operations to reconstruct a primitive type
    from multiple reads.

    Disclaimer: I haven't done thorough speed testing on this since node 0.4,
    just some minor speed testing when Buffers had the readUInt32/etc methods
    added. Also, I think my speed tests were concerned more with writing than
    reading, but it's basically the same code, so it's probably similar results
    ;).
    On Thursday, October 4, 2012 7:27:17 PM UTC-7, Ben Noordhuis wrote:
    On Thu, Oct 4, 2012 at 6:19 PM, Jimb Esser wrote:
    If you want *really* fast buffer access for larger integer types and float
    types, you can set up a native module that calls
    SetIndexedPropertiesToExternalArrayData a few times on your Buffer and gives
    you a native indexed view into the buffer for whatever primitive type you
    want to read (much, much, much faster than trying to read a float
    byte-by-byte and assemble it). We do this in our internal networking
    buffers, and got quite a speed-up compared to combining bytes in JS (though
    either are *way* faster than calling into native methods to do the read,
    except maybe with floats). Note: for un-aligned reads (e.g. reading a
    64-bit float from 3 bytes within a buffer), I think this only works on some
    platforms, but those platforms include x86 and x64, which is all we care
    about in our case.
    You don't have to drop down to C++ for that, node.js supports typed
    arrays[1]. I probably should have mentioned that in my other post. :-)

    [1] https://developer.mozilla.org/en-US/docs/JavaScript_typed_arrays
    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
  • NodeNinja at Oct 5, 2012 at 11:28 am

    On Friday, October 5, 2012 12:29:00 PM UTC+5:30, Jimb Esser wrote:
    The gist of it is just to take a Buffer, for each type you want to read
    (say, e.g, 32-bit ints), make a number of views equal to the possible byte
    offsets (e.g. 4) and then do a read with the right offset from the right
    view (e.g. view[offset % 4][offset >> 2]). V8 does "indexed properties to
    external data" lookups super-fast (Buffers use this), so this ends up being
    much faster than doing the JS operations to reconstruct a primitive type
    from multiple reads.

    Hello Jimb,
    I haven't used typed arrays as yet, so are you making a performance
    comparison of some sorts here between Typed Arrays and Buffers. Is a Typed
    Array at par with Buffer when for reading / writing uint32 or int32
    values?

    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
  • Ben Noordhuis at Oct 5, 2012 at 12:58 pm

    On Fri, Oct 5, 2012 at 8:59 AM, Jimb Esser wrote:
    Ah, yeah, it does now! We wrote our module back on node 0.4, which, I
    think, did not have typed arrays. That being said, it seems TypedArrays do
    not support un-aligned views (e.g. if you're reading a stream and want to
    read a F64 at byte offset 3, it throws an exception), as well they
    shouldn't, as that's not guaranteed to be efficient cross-platform, though
    is critical to performance in our specific situation. Also, it doesn't seem
    you can make a typed array view off of a Buffer (or get an ArrayBuffer off
    of a network read?), so that involves some copying, although since creating
    views is a native call and a bit expensive, we found it quickest to copy any
    network-received Buffer into one that already has views created anyway.
    First-class support for ArrayBuffers and typed arrays in general is
    something we want to add eventually (no fixed date though).

    0.8.x supports creating typed arrays from Buffer objects:

    var buf = new Buffer(16);
    var arr = new Int32Array(buf, 4); // offset, defaults to 0

    The buffer size and offset need to be multiples of the type size (4
    for Int32Array, 8 for a Float64Array, etc.). The typed array is a view
    of the buffer, i.e. it's zero-copy.

    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
  • Dan Milon at Oct 4, 2012 at 8:28 pm
    Curious, shouldn't V8 be able to inline the readUInt8 function call, or
    there's more than that?
    On Thu, Oct 4, 2012 at 1:43 AM, Ben Noordhuis wrote:
    On Thu, Oct 4, 2012 at 1:37 AM, NodeNinja wrote:
    Doing some tests on windows with node v0.8.11
    --------------------------------------------
    var buf = new Buffer(10000000);
    buf.fill(0xFF);

    var len = buf.length;
    var a;
    var start = new Date();
    for(var b = 0 ; b < len; b++){
    a = buf.readUInt8(b);
    }
    var end = new Date();

    console.log('Buffer reads ' + (end - start) + ' ms'); // Prints 81ms

    ------------------------------------------------

    var arr = [];
    var len = 10000000;
    for(b = 0 ; b < len; b++){
    arr[b] = 0xFF;
    }
    var len = arr.length;
    console.log(len);
    start = new Date();
    for(b = 0 ; b < len; b++){
    a = arr[b];
    }
    end = new Date();

    console.log('Array reads ' + (end - start) + ' ms'); // Prints 29ms

    And I always thought that buffers would be faster than arrays in node.js ?
    :(
    Am I doing something wrong ??
    Don't use buf.readUInt8(), it's only there for the sake of parity with
    the other .readUInt*() functions. Replace it with `a = buf[b]` and
    you'll see a 4-5x speed-up.

    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines:
    https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
  • NodeNinja at Oct 4, 2012 at 11:08 pm

    On Friday, October 5, 2012 1:58:17 AM UTC+5:30, Dan Milon wrote:
    Curious, shouldn't V8 be able to inline the readUInt8 function call, or
    there's more than that?

    That's what even I was thinking about if all the buf.read*() and
    the buf.write*() functions are much slower than the indexed version.
    Possibly Ben would know better about this.

    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
  • Ben Noordhuis at Oct 5, 2012 at 2:30 am

    On Fri, Oct 5, 2012 at 1:08 AM, NodeNinja wrote:
    On Friday, October 5, 2012 1:58:17 AM UTC+5:30, Dan Milon wrote:
    Curious, shouldn't V8 be able to inline the readUInt8 function call, or
    there's more than that?
    That's what even I was thinking about if all the buf.read*() and the
    buf.write*() functions are much slower than the indexed version. Possibly
    Ben would know better about this.
    The read*() and write*() functions do more than simple stores and
    loads, they also perform (optional) error checks and byte swaps.
    Besides, they're pure JS functions while buf[i] is an intrinsic. Can't
    compete with that. :-)

    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
  • NodeNinja at Oct 5, 2012 at 11:03 am
    On Friday, October 5, 2012 8:00:45 AM UTC+5:30, Ben Noordhuis wrote:

    You don't have to drop down to C++ for that, node.js supports typed
    arrays[1]. I probably should have mentioned that in my other post. :-)

    Thats some good news! :)


    Besides, they're pure JS functions while buf[i] is an intrinsic.
    Can't compete with that. :-)

    Is there a comprehensive list of what functions are intrinsic and which
    functions are in pure JS?


    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en
  • Ben Noordhuis at Oct 5, 2012 at 1:07 pm

    On Fri, Oct 5, 2012 at 1:03 PM, NodeNinja wrote:
    Is there a comprehensive list of what functions are intrinsic and which
    functions are in pure JS?
    Basically anything that's not an operator is a function. That's also
    true for JS built-ins like Array.prototype.sort(), etc.

    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nodejs@googlegroups.com
    To unsubscribe from this group, send email to
    nodejs+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupnodejs @
categoriesnodejs
postedOct 3, '12 at 11:37p
activeOct 5, '12 at 1:07p
posts16
users4
websitenodejs.org
irc#node.js

People

Translate

site design / logo © 2022 Grokbase