FAQ
I needed a library to handle marshaling application defined (C
struct) binary TCP/IP protocol packets in a Clojure client app
replacing a legacy C++ client and there didn't seem to be an existing
library that performed as I needed. So, I've written a Marshal library
that marshals from the network to/from Clojure.

Github: http://github.com/russellc/Marshal
Clojars: https://clojars.org/marshal

//C header file definition
struct packet {
unsigned long type;
unsigned long size;
long data[1];
};

(require '[marshal.core :as m])
(def packet (m/struct :type m/uint32 :size m/uint32 :data (m/array m/
sint32 :size)))
(m/write output-stream packet {:type 1 :size 2 :data [1 -1]})
(m/read input-stream packet)
=> {:type 1 :size 2 :data [1 -1]}

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Search Discussions

  • Alan Malloy at Feb 5, 2012 at 11:10 pm
    Zack Tellman's library Gloss is excellent for this, if a bit confusing
    to understand at first.
    On Feb 5, 2:50 pm, russellc wrote:
    I needed a library  to handle marshaling application defined (C
    struct) binary TCP/IP protocol packets in a Clojure client app
    replacing a legacy C++ client and there didn't seem to be an existing
    library that performed as I needed. So, I've written a Marshal library
    that marshals from the network to/from Clojure.

    Github:http://github.com/russellc/Marshal
    Clojars:https://clojars.org/marshal

    //C header file definition
    struct packet {
    unsigned long type;
    unsigned long size;
    long data[1];

    };

    (require '[marshal.core :as m])
    (def packet (m/struct :type m/uint32 :size m/uint32 :data (m/array m/
    sint32 :size)))
    (m/write output-stream packet {:type 1 :size 2 :data [1 -1]})
    (m/read input-stream packet)
    => {:type 1 :size 2 :data [1 -1]}
    --
    You received this message because you are subscribed to the Google
    Groups "Clojure" group.
    To post to this group, send email to clojure@googlegroups.com
    Note that posts from new members are moderated - please be patient with your first post.
    To unsubscribe from this group, send email to
    clojure+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/clojure?hl=en
  • Russell Christopher at Feb 6, 2012 at 12:09 am
    I did look at Gloss before writing Marshal and decided against using it in
    our application suite, I wanted something more focused utilizing
    InputStream/OutputStream interface and able to handle variable sized
    arrays/strings (size of array or string specified in packet).

    On Sun, Feb 5, 2012 at 6:10 PM, Alan Malloy wrote:

    Zack Tellman's library Gloss is excellent for this, if a bit confusing
    to understand at first.
    On Feb 5, 2:50 pm, russellc wrote:
    I needed a library to handle marshaling application defined (C
    struct) binary TCP/IP protocol packets in a Clojure client app
    replacing a legacy C++ client and there didn't seem to be an existing
    library that performed as I needed. So, I've written a Marshal library
    that marshals from the network to/from Clojure.

    Github:http://github.com/russellc/Marshal
    Clojars:https://clojars.org/marshal

    //C header file definition
    struct packet {
    unsigned long type;
    unsigned long size;
    long data[1];

    };

    (require '[marshal.core :as m])
    (def packet (m/struct :type m/uint32 :size m/uint32 :data (m/array m/
    sint32 :size)))
    (m/write output-stream packet {:type 1 :size 2 :data [1 -1]})
    (m/read input-stream packet)
    => {:type 1 :size 2 :data [1 -1]}
    --
    You received this message because you are subscribed to the Google
    Groups "Clojure" group.
    To post to this group, send email to clojure@googlegroups.com
    Note that posts from new members are moderated - please be patient with
    your first post.
    To unsubscribe from this group, send email to
    clojure+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/clojure?hl=en
    --
    You received this message because you are subscribed to the Google
    Groups "Clojure" group.
    To post to this group, send email to clojure@googlegroups.com
    Note that posts from new members are moderated - please be patient with your first post.
    To unsubscribe from this group, send email to
    clojure+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/clojure?hl=en
  • Alan Malloy at Feb 6, 2012 at 1:01 am
    Gloss certainly handles variable-size frames: the wiki even has a
    number of examples of the different built-in ways you can encode the
    size, and you can write a custom size-detector if none of those suit
    you. As for streams vs channels, my understanding is that the java.nio
    package Gloss uses was introduced as a higher-performance option
    compared to java.io. Like most people, I don't think I've ever really
    understood nio, but I suspect that Zack does and chose it for a
    reason.

    My point is that Gloss is a fantastic library focused at exactly your
    problem domain, and you should probably do some more investigating
    before dismissing it as unsuitable (I infer you haven't done much
    research because you think it isn't focused and that it's not possible/
    easy to handle variable-length arrays).

    For example, here's how to do your example with Gloss:

    (def codec (compile-frame {:type :uint32 :data
    (repeated :uint32 :prefix :uint32)}))

    (encode codec {:type 3 :data [1 2 3]})
    ;=> (#<HeapByteBuffer java.nio.HeapByteBuffer[pos=0 lim=16 cap=16]>
    #<HeapByteBuffer java.nio.HeapByteBuffer[pos=0 lim=4 cap=4]>)

    (decode codec *1)
    ;=> {:type 3, :data [1 2 3]}

    If you really-really want the :size to be present in the Clojure data
    structures (rather than just ignoring it as an artifact of the
    protocol), you can do that by messing with a header in the codec. You
    can also make type an enum instead of just an int32, so that Gloss
    converts it into a meaningful keyword for you instead of having to
    deal with the number yourself. I ignored both of these issues for
    simplicity and due to my ignorance of your actual protocol.

    Perhaps more importantly than any of this, I suspect you're relying on
    the maps you use to retain their order - that is, you assume because
    you specify type, then size, then data, they will be encoded in that
    order. But maps are not sorted; it's only a coincidence that for small
    maps, on the current version of Clojure, they happen to be in the
    order you specified them. If you encoded a structure with ten or more
    fields, your library would suddenly break because the fields would be
    ordered wrong. For regular maps, Gloss sorts the maps alphabetically
    by key - in my example it encoded data before type. This is consistent
    across versions of Clojure and all map sizes (great!) but not what you
    actually want when you interoperate with C. So instead, Gloss allows
    you to give it a vector of pairs (or something like that), or a
    special map that promises to retain the order you specify.

    This is an example of the sort of tricky edge-case (or scalability
    issue) that would surely slip by me when implementing my own binary-
    encoding library, but which has already been addressed by Gloss. By
    choosing a mature and well-known library instead of making your own,
    you get more features and fewer bugs.
    On Feb 5, 4:09 pm, Russell Christopher wrote:
    I did look at Gloss before writing Marshal and decided against using it in
    our application suite, I wanted something more focused utilizing
    InputStream/OutputStream interface and able to handle variable sized
    arrays/strings (size of array or string specified in packet).
    On Sun, Feb 5, 2012 at 6:10 PM, Alan Malloy wrote:
    Zack Tellman's library Gloss is excellent for this, if a bit confusing
    to understand at first.
    On Feb 5, 2:50 pm, russellc wrote:
    I needed a library  to handle marshaling application defined (C
    struct) binary TCP/IP protocol packets in a Clojure client app
    replacing a legacy C++ client and there didn't seem to be an existing
    library that performed as I needed. So, I've written a Marshal library
    that marshals from the network to/from Clojure.
    Github:http://github.com/russellc/Marshal
    Clojars:https://clojars.org/marshal
    //C header file definition
    struct packet {
    unsigned long type;
    unsigned long size;
    long data[1];
    };
    (require '[marshal.core :as m])
    (def packet (m/struct :type m/uint32 :size m/uint32 :data (m/array m/
    sint32 :size)))
    (m/write output-stream packet {:type 1 :size 2 :data [1 -1]})
    (m/read input-stream packet)
    => {:type 1 :size 2 :data [1 -1]}
    --
    You received this message because you are subscribed to the Google
    Groups "Clojure" group.
    To post to this group, send email to clojure@googlegroups.com
    Note that posts from new members are moderated - please be patient with
    your first post.
    To unsubscribe from this group, send email to
    clojure+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/clojure?hl=en
    --
    You received this message because you are subscribed to the Google
    Groups "Clojure" group.
    To post to this group, send email to clojure@googlegroups.com
    Note that posts from new members are moderated - please be patient with your first post.
    To unsubscribe from this group, send email to
    clojure+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/clojure?hl=en
  • Russell Christopher at Feb 6, 2012 at 1:21 am
    Take it easy, no disrespect intended to Gloss, it just wasn't my preference
    that is all.
    You're incorrect about Marshal relying on maps to retain their order!

    On Sun, Feb 5, 2012 at 8:01 PM, Alan Malloy wrote:

    Gloss certainly handles variable-size frames: the wiki even has a
    number of examples of the different built-in ways you can encode the
    size, and you can write a custom size-detector if none of those suit
    you. As for streams vs channels, my understanding is that the java.nio
    package Gloss uses was introduced as a higher-performance option
    compared to java.io. Like most people, I don't think I've ever really
    understood nio, but I suspect that Zack does and chose it for a
    reason.

    My point is that Gloss is a fantastic library focused at exactly your
    problem domain, and you should probably do some more investigating
    before dismissing it as unsuitable (I infer you haven't done much
    research because you think it isn't focused and that it's not possible/
    easy to handle variable-length arrays).

    For example, here's how to do your example with Gloss:

    (def codec (compile-frame {:type :uint32 :data
    (repeated :uint32 :prefix :uint32)}))

    (encode codec {:type 3 :data [1 2 3]})
    ;=> (#<HeapByteBuffer java.nio.HeapByteBuffer[pos=0 lim=16 cap=16]>
    #<HeapByteBuffer java.nio.HeapByteBuffer[pos=0 lim=4 cap=4]>)

    (decode codec *1)
    ;=> {:type 3, :data [1 2 3]}

    If you really-really want the :size to be present in the Clojure data
    structures (rather than just ignoring it as an artifact of the
    protocol), you can do that by messing with a header in the codec. You
    can also make type an enum instead of just an int32, so that Gloss
    converts it into a meaningful keyword for you instead of having to
    deal with the number yourself. I ignored both of these issues for
    simplicity and due to my ignorance of your actual protocol.

    Perhaps more importantly than any of this, I suspect you're relying on
    the maps you use to retain their order - that is, you assume because
    you specify type, then size, then data, they will be encoded in that
    order. But maps are not sorted; it's only a coincidence that for small
    maps, on the current version of Clojure, they happen to be in the
    order you specified them. If you encoded a structure with ten or more
    fields, your library would suddenly break because the fields would be
    ordered wrong. For regular maps, Gloss sorts the maps alphabetically
    by key - in my example it encoded data before type. This is consistent
    across versions of Clojure and all map sizes (great!) but not what you
    actually want when you interoperate with C. So instead, Gloss allows
    you to give it a vector of pairs (or something like that), or a
    special map that promises to retain the order you specify.

    This is an example of the sort of tricky edge-case (or scalability
    issue) that would surely slip by me when implementing my own binary-
    encoding library, but which has already been addressed by Gloss. By
    choosing a mature and well-known library instead of making your own,
    you get more features and fewer bugs.
    On Feb 5, 4:09 pm, Russell Christopher wrote:
    I did look at Gloss before writing Marshal and decided against using it in
    our application suite, I wanted something more focused utilizing
    InputStream/OutputStream interface and able to handle variable sized
    arrays/strings (size of array or string specified in packet).
    On Sun, Feb 5, 2012 at 6:10 PM, Alan Malloy wrote:
    Zack Tellman's library Gloss is excellent for this, if a bit confusing
    to understand at first.
    On Feb 5, 2:50 pm, russellc wrote:
    I needed a library to handle marshaling application defined (C
    struct) binary TCP/IP protocol packets in a Clojure client app
    replacing a legacy C++ client and there didn't seem to be an existing
    library that performed as I needed. So, I've written a Marshal
    library
    that marshals from the network to/from Clojure.
    Github:http://github.com/russellc/Marshal
    Clojars:https://clojars.org/marshal
    //C header file definition
    struct packet {
    unsigned long type;
    unsigned long size;
    long data[1];
    };
    (require '[marshal.core :as m])
    (def packet (m/struct :type m/uint32 :size m/uint32 :data (m/array m/
    sint32 :size)))
    (m/write output-stream packet {:type 1 :size 2 :data [1 -1]})
    (m/read input-stream packet)
    => {:type 1 :size 2 :data [1 -1]}
    --
    You received this message because you are subscribed to the Google
    Groups "Clojure" group.
    To post to this group, send email to clojure@googlegroups.com
    Note that posts from new members are moderated - please be patient with
    your first post.
    To unsubscribe from this group, send email to
    clojure+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/clojure?hl=en
    --
    You received this message because you are subscribed to the Google
    Groups "Clojure" group.
    To post to this group, send email to clojure@googlegroups.com
    Note that posts from new members are moderated - please be patient with
    your first post.
    To unsubscribe from this group, send email to
    clojure+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/clojure?hl=en
    --
    You received this message because you are subscribed to the Google
    Groups "Clojure" group.
    To post to this group, send email to clojure@googlegroups.com
    Note that posts from new members are moderated - please be patient with your first post.
    To unsubscribe from this group, send email to
    clojure+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/clojure?hl=en

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupclojure @
categoriesclojure
postedFeb 5, '12 at 10:50p
activeFeb 6, '12 at 1:21a
posts5
users2
websiteclojure.org
irc#clojure

People

Translate

site design / logo © 2022 Grokbase