FAQ
Hi to everyone!

I've developed a memory mapped implementation of SequenceFile to be used
inside Artemis's Journal and i've encountered a big question: can i use
sun.misc.Unsafe?

I've noticed that between all the dependencies of Artemis there was Netty's
libs with its PlatformDependent class and i've finally choose to use it
instead of "naked" Unsafe,but i'm not completly satisfied by this solution
because PlatformDependent does not provide all the methods of Unsafe (e.g.
Unsafe::pageSize).
With such methods there are a lot of improvements and features that can be
developed, considering that in the future of Java there will be a place for
a safer Unsafe :)
There exists a good policy for all that cases in which something so "unsafe"
and "platform dependend" is needed?What the other devs think about it?


With Regards,
Francesco





--
View this message in context: http://activemq.2283324.n4.nabble.com/sun-misc-Unsafe-VS-io-netty-util-internal-PlatformDependent-VS-null-tp4712785.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.

Search Discussions

  • Clebert Suconic at Jun 9, 2016 at 6:20 pm
    I would isolate it on a library. Netty does have it but it's well
    controlled on a single place I think (easier to maintain it)

    I had played with memory mapped years ago and I've decided to use
    libaio instead. Did you run any micro benchmarks, have you made any
    progress? it seems like a nice idea if you found a breakthrough on
    making it performant.. I'm quite interested on it.
    On Wed, Jun 8, 2016 at 10:52 AM, nigro_franz wrote:
    Hi to everyone!

    I've developed a memory mapped implementation of SequenceFile to be used
    inside Artemis's Journal and i've encountered a big question: can i use
    sun.misc.Unsafe?

    I've noticed that between all the dependencies of Artemis there was Netty's
    libs with its PlatformDependent class and i've finally choose to use it
    instead of "naked" Unsafe,but i'm not completly satisfied by this solution
    because PlatformDependent does not provide all the methods of Unsafe (e.g.
    Unsafe::pageSize).
    With such methods there are a lot of improvements and features that can be
    developed, considering that in the future of Java there will be a place for
    a safer Unsafe :)
    There exists a good policy for all that cases in which something so "unsafe"
    and "platform dependend" is needed?What the other devs think about it?


    With Regards,
    Francesco





    --
    View this message in context: http://activemq.2283324.n4.nabble.com/sun-misc-Unsafe-VS-io-netty-util-internal-PlatformDependent-VS-null-tp4712785.html
    Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.


    --
    Clebert Suconic
  • Clebert Suconic at Jun 9, 2016 at 6:21 pm
    Like with Artemis, there's the Artemis-native, where I isolate the use
    of the native module.

    Maybe you could create a project to just control MapFiles.. it would
    be easier to maintain it I think.

    On Thu, Jun 9, 2016 at 2:20 PM, Clebert Suconic
    wrote:
    I would isolate it on a library. Netty does have it but it's well
    controlled on a single place I think (easier to maintain it)

    I had played with memory mapped years ago and I've decided to use
    libaio instead. Did you run any micro benchmarks, have you made any
    progress? it seems like a nice idea if you found a breakthrough on
    making it performant.. I'm quite interested on it.
    On Wed, Jun 8, 2016 at 10:52 AM, nigro_franz wrote:
    Hi to everyone!

    I've developed a memory mapped implementation of SequenceFile to be used
    inside Artemis's Journal and i've encountered a big question: can i use
    sun.misc.Unsafe?

    I've noticed that between all the dependencies of Artemis there was Netty's
    libs with its PlatformDependent class and i've finally choose to use it
    instead of "naked" Unsafe,but i'm not completly satisfied by this solution
    because PlatformDependent does not provide all the methods of Unsafe (e.g.
    Unsafe::pageSize).
    With such methods there are a lot of improvements and features that can be
    developed, considering that in the future of Java there will be a place for
    a safer Unsafe :)
    There exists a good policy for all that cases in which something so "unsafe"
    and "platform dependend" is needed?What the other devs think about it?


    With Regards,
    Francesco





    --
    View this message in context: http://activemq.2283324.n4.nabble.com/sun-misc-Unsafe-VS-io-netty-util-internal-PlatformDependent-VS-null-tp4712785.html
    Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.


    --
    Clebert Suconic


    --
    Clebert Suconic
  • Nigro_franz at Jun 10, 2016 at 8:10 am
    Hi!

    Thanks for the good advice, i'm happy to let Unsafe improve further the
    performances of this mapped implementation
    Here https://issues.apache.org/jira/browse/ARTEMIS-508
    <https://issues.apache.org/jira/browse/ARTEMIS-508> there is the JIRA
    related to the memory mapped implementation of SequenceFile: i need to
    squash the (too many) commits into one before doing any PR and obviosly
    reimplement a "confined" version of PlatformDependent that let Android too
    to use the mmap implementation (crossing the fingers, ART VM is pure evil).
    I've built for now only 2 benchmarks (latency focused), but without using
    JHM because of the coordinated omission issue, thus i've found JLBH (Java
    Latency Benchmark Harness)
    http://www.rationaljava.com/2016/04/a-series-of-posts-on-jlbh-java-latency.html
    <http://www.rationaljava.com/2016/04/a-series-of-posts-on-jlbh-java-latency.html>
    that is best suited for this kind of measurements IMHO.
    The results of the benchmarks are pretty good against NIO (if you push data
    in "burst" of less then 10% of your memory, for a default configured latest
    ubuntu linux box): about 2 order of magnitude less in the 99 %tile latency
    measurements.
    With these benchs i could measure how much on different configured HW the
    mmap Sequence File could "keep the pace" with a sustained load of messages,
    thus producing something pretty similar to a real use case eg without any
    "math benefits" derived by the backpressure of the OS during the page
    faulting unavoidable pauses.
    I've planned to solve the Page Faulting issue using a pure async
    implementation, using a single producer single consumer wait-free ringBuffer
    that feeds an event loop responsible to write the data on the MappedFile but
    the benchs that i've built (around Journal and SequenceFile) show that the
    higher layers (above the mmap impl of SequenceFile) slow even this pure-sync
    implementation, so i'm not sure that the Journal could receive a perceptible
    benefit from an async one considering that it will only reduce the hiccups
    when a page fault of few microseconds occour (anyway it could be tuned with
    Huge Pages and increasing the dirty ratio).
    I'm very curious to test it against libaio and i will be very happy if you
    are interested to try it with me, in order to be objective: my expectations
    are that with burst of small messages the mmap impl will be faster (crossing
    JNI has higher cost than call an intrinsified tiny copy with Unsafe, in my
    experience) , the opposite while the size of the messages grows. If you need
    other benchs (throughput focused with JHM) or need more info about the
    implementation i'm ever available



    --
    View this message in context: http://activemq.2283324.n4.nabble.com/sun-misc-Unsafe-VS-io-netty-util-internal-PlatformDependent-VS-null-tp4712785p4712866.html
    Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categoriesactivemq
postedJun 8, '16 at 3:30p
activeJun 10, '16 at 8:10a
posts4
users2
websiteactivemq.apache.org

2 users in discussion

Nigro_franz: 2 posts Clebert Suconic: 2 posts

People

Translate

site design / logo © 2018 Grokbase