Grokbase › Groups › Go › golang-nuts › December 2013
FAQ
Hi,

I work on a large Go project implementing a XMPP [1] chat server. One of
our biggest problem is handling XML in a clean way [3]. I'm currently
trying to improve our approach to the problem, and writing this message to
seek advise from the community. Some of this may seem negative, but I'm
genuinely happy with Go overall, and also interested in contributing to
making things better.

The Marshal/Unmarshal interface of encoding/xml is at fundamental odds with
the XMPP protocol. Unmarshal/Marshal expect XML that can be mapped to a
static structure [2]. XMPP requires processing of XML that may have a
static structure at its core, but can be extended with arbitrary elements
and attributes by servers and clients alike [10].

So for all practical purposes, XMPP requires a DOM [9] / Tree kind of data
structure. Additionally XPath is really nice to have (but one can do
without).

And this is where things get tricky. There currently doesn't seem to be a
mature DOM implementation for Go. xmldom-go [4] comes close, but has seen
no commits for 9 months and looking at the code, I suspect it would take a
lot of effort to finish it and clean it up. So at this point the only
mature solution seems to be libxml2 via gokogiri [5], which is what we
ended up with so far. However it has two major problems:

1. It introduces manual memory management into a garbage collected
environment. While Go makes it easy to create C bindings, they're
incredibly annoying to use in practice because of this. SetFinalizer looks
like it could help, but the guarantees it provides are currently too weak,
and it's my understanding that there are no plans to change that.
2. libxml2 has a really nice API for parsing streaming XML, but it doesn't
really work [6] and I've had no luck getting the maintainers attention for
the patch I provided [7].

The end result of these two problems is that we're forced to do manual
memory management (which has lead to bugs causing segfaults, panics,
infinite loops and data corruption) as well as using encoding/xml to buffer
up individual XML stanzas before parsing them for a second time with
gokogiri : (.

Additionally we've also been unable to use encoding/xml in more simplistic
marshal scenarios involving namespaces, but maybe this is just our
inability to use the API correctly [8]?

Anyway, at this point we're between a rock and a hard place when it comes
to handling xml. It seems like implementing a simple DOM pkg + support for
basic XPath for our internal needs may be the best solution for now, but
that's not going to make things any better for the community, as we'd
rather not release another incomplete/ad-hoc solution into the ecosystem.

So I'm wondering if anybody has advise. Are there plans of adding a
DOM-style API to the go core? If yes, I'd like to help. If no, does anybody
see any feasible alternatives for our situation? [11]

Many thanks in advance,
Felix Geisendörfer

[1]: http://xmpp.org/xmpp-protocols/
[2]: I use the term "static structure" to describe compile-time defined
structures made up of structs and slices, rather than a Tree/DOM-like data
structure.
[3]: Have a drink if you laughed, otherwise continue on.
[4]: http://godoc.org/bitbucket.org/rj/xmldom-go
[5]: http://godoc.org/github.com/moovweb/gokogiri
[6]: https://mail.gnome.org/archives/xml/2013-July/msg00015.html
[7]: https://mail.gnome.org/archives/xml/2013-August/msg00025.html
[8]: http://play.golang.org/p/RxnOZ1-I8E
[9]: http://www.w3.org/TR/dom/
[10]: The complexity of XMPP is worth a discussion as well, but we're stuck
with it for this project.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Mateusz Czapliński at Dec 13, 2013 at 11:25 am ⇧

    On Friday, December 13, 2013 11:40:52 AM UTC+1, Felix Geisendoerfer wrote:
    So for all practical purposes, XMPP requires a DOM [9] / Tree kind of
    data structure. Additionally XPath is really nice to have (but one can do
    without).
    Sorry if you've already evaluated it, but you don't seem to mention, so
    maybe you might want to check if "x2j" wouldn't help you by chance:
       https://github.com/clbanning/x2j
    I didn't really use it yet, so no idea how well it works in practice and if
    it has all features you need, but to me it looked interesting at first
    sight. The general idea from what I understand is that the package decodes
    XML to nested map[string]interface{} "tree". So not "real DOM", but some
    kinda tree-like something, and also seems to have some (really basic)
    "XPath-like" in "func MapValue(...)": "'path' is a period-separated
    hierarchy of keys in the map."

    /Mateusz.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Caleb Doxsey at Dec 13, 2013 at 4:22 pm ⇧
    Hey Felix,

    Though not a DOM the encoding/xml library does support a streaming
    interface:

    http://play.golang.org/p/tHa5A8JFp5

    It's a bit of work, and you won't get XPath, but depending on your needs it
    may actually be better. A streaming API can be much faster than one that
    builds up an entire DOM in memory.
    On Friday, December 13, 2013 3:40:52 AM UTC-7, Felix Geisendoerfer wrote:

    Hi,

    I work on a large Go project implementing a XMPP [1] chat server. One of
    our biggest problem is handling XML in a clean way [3]. I'm currently
    trying to improve our approach to the problem, and writing this message to
    seek advise from the community. Some of this may seem negative, but I'm
    genuinely happy with Go overall, and also interested in contributing to
    making things better.

    The Marshal/Unmarshal interface of encoding/xml is at fundamental odds
    with the XMPP protocol. Unmarshal/Marshal expect XML that can be mapped to
    a static structure [2]. XMPP requires processing of XML that may have a
    static structure at its core, but can be extended with arbitrary elements
    and attributes by servers and clients alike [10].

    So for all practical purposes, XMPP requires a DOM [9] / Tree kind of
    data structure. Additionally XPath is really nice to have (but one can do
    without).

    And this is where things get tricky. There currently doesn't seem to be a
    mature DOM implementation for Go. xmldom-go [4] comes close, but has seen
    no commits for 9 months and looking at the code, I suspect it would take a
    lot of effort to finish it and clean it up. So at this point the only
    mature solution seems to be libxml2 via gokogiri [5], which is what we
    ended up with so far. However it has two major problems:

    1. It introduces manual memory management into a garbage collected
    environment. While Go makes it easy to create C bindings, they're
    incredibly annoying to use in practice because of this. SetFinalizer looks
    like it could help, but the guarantees it provides are currently too weak,
    and it's my understanding that there are no plans to change that.
    2. libxml2 has a really nice API for parsing streaming XML, but it doesn't
    really work [6] and I've had no luck getting the maintainers attention for
    the patch I provided [7].

    The end result of these two problems is that we're forced to do manual
    memory management (which has lead to bugs causing segfaults, panics,
    infinite loops and data corruption) as well as using encoding/xml to buffer
    up individual XML stanzas before parsing them for a second time with
    gokogiri : (.

    Additionally we've also been unable to use encoding/xml in more simplistic
    marshal scenarios involving namespaces, but maybe this is just our
    inability to use the API correctly [8]?

    Anyway, at this point we're between a rock and a hard place when it comes
    to handling xml. It seems like implementing a simple DOM pkg + support for
    basic XPath for our internal needs may be the best solution for now, but
    that's not going to make things any better for the community, as we'd
    rather not release another incomplete/ad-hoc solution into the ecosystem.

    So I'm wondering if anybody has advise. Are there plans of adding a
    DOM-style API to the go core? If yes, I'd like to help. If no, does anybody
    see any feasible alternatives for our situation? [11]

    Many thanks in advance,
    Felix Geisendörfer

    [1]: http://xmpp.org/xmpp-protocols/
    [2]: I use the term "static structure" to describe compile-time defined
    structures made up of structs and slices, rather than a Tree/DOM-like data
    structure.
    [3]: Have a drink if you laughed, otherwise continue on.
    [4]: http://godoc.org/bitbucket.org/rj/xmldom-go
    [5]: http://godoc.org/github.com/moovweb/gokogiri
    [6]: https://mail.gnome.org/archives/xml/2013-July/msg00015.html
    [7]: https://mail.gnome.org/archives/xml/2013-August/msg00025.html
    [8]: http://play.golang.org/p/RxnOZ1-I8E
    [9]: http://www.w3.org/TR/dom/
    [10]: The complexity of XMPP is worth a discussion as well, but we're
    stuck with it for this project.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Felix Geisendoerfer at Dec 13, 2013 at 4:34 pm ⇧

    On Friday, December 13, 2013 5:22:34 PM UTC+1, Caleb Doxsey wrote:
    Though not a DOM the encoding/xml library does support a streaming
    interface:

    http://play.golang.org/p/tHa5A8JFp5

    It's a bit of work, and you won't get XPath, but depending on your needs
    it may actually be better. A streaming API can be much faster than one that
    builds up an entire DOM in memory.
    Thanks for coming up with this suggestion!

    Unfortunately this doesn't solve the core problem: XMPP allows clients to
    add custom nodes and attributes to any stanza. So a XMPP server has to
    process stanzas in a way that does not loose this data as it often needs to
    be forwarded to another entity (client or server) in its original form.

    Your example could of course be extended in a way to handle that, but now
    you'll end up with a data structure that is a mix between a struct and a
    tree, which will become problematic when trying to serialize the data again.

    --fg

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Wkharold at Dec 13, 2013 at 4:53 pm ⇧
    I have no solution to your problem, but I feel your pain. The crux of the
    matter is that XML was a phenomenally bad choice of encoding for a wire
    protocol. Per, http://harmful.cat-v.org/software, IRC and STOMP are better
    choices for messaging. BEEP based APEX protocol, which did include a small,
    manageable, dollop of XML, would have been a superior choice as well. Best
    of luck with XMPP.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Will Norris at Dec 13, 2013 at 4:55 pm ⇧

    On Fri, Dec 13, 2013 at 8:34 AM, Felix Geisendoerfer wrote:
    On Friday, December 13, 2013 5:22:34 PM UTC+1, Caleb Doxsey wrote:

    Though not a DOM the encoding/xml library does support a streaming
    interface:

    http://play.golang.org/p/tHa5A8JFp5

    It's a bit of work, and you won't get XPath, but depending on your needs
    it may actually be better. A streaming API can be much faster than one that
    builds up an entire DOM in memory.
    Thanks for coming up with this suggestion!

    Unfortunately this doesn't solve the core problem: XMPP allows clients to
    add custom nodes and attributes to any stanza. So a XMPP server has to
    process stanzas in a way that does not loose this data as it often needs to
    be forwarded to another entity (client or server) in its original form.

    Your example could of course be extended in a way to handle that, but now
    you'll end up with a data structure that is a mix between a struct and a
    tree, which will become problematic when trying to serialize the data again.
    I believe that's what the go protobuf library (
    https://code.google.com/p/goprotobuf/) does, using it's "XXX_unrecognized"
    field. So there is at least some precedent for doing that, and hopefully
    some useful code in there.

    --fg

    --
    You received this message because you are subscribed to the Google Groups
    "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Gary b at Dec 13, 2013 at 5:31 pm ⇧
    I wrote an XMPP server in another language long ago. If I had to write a
    server in Go, I would do the following:

    - Represent XML using a structure similar to Python's element tree:

        type Element struct {
             Name xml.Name
             Attr map[xml.Name]string
             Elements []*Element // Children
             Text string // First text child.
             Tail string // Text following this element in the parent.
        }

        It's easy to write most of what an XMPP server needs to do with this
        representation and a few helper methods.

    - Parse stanzas from the connection to the Element type using
       http://godoc.org/encoding/xml#Decoder. This should be no more
       than 100 lines of code.

    - Write custom code to encode an Element tree as XML text. This
       should be no more than 200 lines of code. The biggest complication
       in this code is managing namespace prefixes and declarations.
       Namespaces declared on the root element should not be duplicated
       in stanzas, the stanza elements should be in the default namespace
       and namespaces used by more than one child should be declared in
       the parent to save space.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Felix Geisendoerfer at Dec 13, 2013 at 5:55 pm ⇧

    On Friday, December 13, 2013 6:31:39 PM UTC+1, gary b wrote:
    I wrote an XMPP server in another language long ago. If I had to write a
    server in Go, I would do the following:

    - Represent XML using a structure similar to Python's element tree:

    type Element struct {
    Name xml.Name
    Attr map[xml.Name]string
    Elements []*Element // Children
    Text string // First text child.
    Tail string // Text following this element in the parent.
    }

    It's easy to write most of what an XMPP server needs to do with this
    representation and a few helper methods.

    - Parse stanzas from the connection to the Element type using
    http://godoc.org/encoding/xml#Decoder. This should be no more
    than 100 lines of code.

    - Write custom code to encode an Element tree as XML text. This
    should be no more than 200 lines of code. The biggest complication
    in this code is managing namespace prefixes and declarations.
    Namespaces declared on the root element should not be duplicated
    in stanzas, the stanza elements should be in the default namespace
    and namespaces used by more than one child should be declared in
    the parent to save space.
    Thanks for the detailed suggestion. This is along the lines of what I
    expect we'll end up with. The biggest challenge is probably going to be
    XPath which we use throughout the code base. So we either need to implement
    a minimal subset of that, or convert all queries to explicit tree traversal.

    Anyway, for now I'm moving all our XML handling into one package and am
    writing tests for it. Once that is done I'll experiment with a custom tree
    structure like the one you suggested.

    I'll report back once I have the results of those attempts.

    That being said, if anybody has some additional ideas, I'd still love to
    hear them. One thing that crossed my mind was to build a simple garbage
    collector that tracks all pointers to libxml2 and scans the heap for
    references to them, but since I've never attempted anything like this, I'd
    rather not go there : ).

    --fg

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • C Banning at Dec 13, 2013 at 6:04 pm ⇧
    It a slightly more generic manner, that's what x2j.ToTree() - or
    x2j.DocToTree() - does; just export the Node members. (The parsing
    function is 64 lines.) It's a lot faster to extract values, etc., however,
    from a map[string]interface{} value than a parse tree.

    xmldom-go has a parse tree, also.
    On Friday, December 13, 2013 11:31:39 AM UTC-6, gary b wrote:

    I wrote an XMPP server in another language long ago. If I had to write a
    server in Go, I would do the following:

    - Represent XML using a structure similar to Python's element tree:

    type Element struct {
    Name xml.Name
    Attr map[xml.Name]string
    Elements []*Element // Children
    Text string // First text child.
    Tail string // Text following this element in the parent.
    }

    It's easy to write most of what an XMPP server needs to do with this
    representation and a few helper methods.

    - Parse stanzas from the connection to the Element type using
    http://godoc.org/encoding/xml#Decoder. This should be no more
    than 100 lines of code.

    - Write custom code to encode an Element tree as XML text. This
    should be no more than 200 lines of code. The biggest complication
    in this code is managing namespace prefixes and declarations.
    Namespaces declared on the root element should not be duplicated
    in stanzas, the stanza elements should be in the default namespace
    and namespaces used by more than one child should be declared in
    the parent to save space.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Codeprojpub at Dec 13, 2013 at 11:24 pm ⇧
    Try this https://github.com/beevik/etree. Relatively small library but it
    did help me with custom XML parsing and creating documents. I wonder no one
    mentioned it.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Felix Geisendoerfer at Dec 14, 2013 at 11:05 am ⇧

    On Fri, Dec 13, 2013 at 6:21 PM, wrote:

    Try this https://github.com/beevik/etree. Relatively small library but it
    did help me with custom XML parsing and creating documents. I wonder no one
    mentioned it.
    Thanks - this looks rather nice indeed. Namespace support seems to be
    missing, but it could be a good starting point!

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Beevik at Dec 25, 2013 at 10:46 pm ⇧
    I just updated the etree package at https://github.com/beevik/etree to
    support namespaces. (The change shouldn't break any existing code.)
    On Saturday, December 14, 2013 3:04:55 AM UTC-8, Felix Geisendoerfer wrote:
    On Fri, Dec 13, 2013 at 6:21 PM, <codep...@gmail.com <javascript:>> wrote:

    Try this https://github.com/beevik/etree. Relatively small library but
    it did help me with custom XML parsing and creating documents. I wonder no
    one mentioned it.
    Thanks - this looks rather nice indeed. Namespace support seems to be
    missing, but it could be a good starting point!
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Felix Geisendoerfer at Dec 27, 2013 at 11:19 am ⇧

    On Wed, Dec 25, 2013 at 11:46 PM, beevik wrote:

    I just updated the etree package at https://github.com/beevik/etree to
    support namespaces. (The change shouldn't break any existing code.)
    Excellent! I've now moved all our XML processing into a single package, so
    I'll make an attempt to replace gokogiri soon. Your library looks very
    promising, so I'll let you know how it goes / send patches if I come across
    problems!

    Cheers,
    --
    Felix Geisendörfer (felixge.de)
    Co-Founder, Debuggable Limited (debuggable.com <http://www.debuggable.com>)
    Mobile: +49 (151) 14967823

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Exa Newspace at Nov 28, 2014 at 3:42 pm ⇧
    Is this project still ongoing Felix ? I was searching for XMPP server in Go
    and did not find anything but that discussion...

    Le vendredi 27 décembre 2013 15:18:44 UTC+4, Felix Geisendoerfer a écrit :

    On Wed, Dec 25, 2013 at 11:46 PM, beevik <br...@beevik.com <javascript:>>
    wrote:
    I just updated the etree package at https://github.com/beevik/etree to
    support namespaces. (The change shouldn't break any existing code.)
    Excellent! I've now moved all our XML processing into a single package, so
    I'll make an attempt to replace gokogiri soon. Your library looks very
    promising, so I'll let you know how it goes / send patches if I come across
    problems!

    Cheers,
    --
    Felix Geisendörfer (felixge.de)
    Co-Founder, Debuggable Limited (debuggable.com <http://www.debuggable.com>
    )
    Mobile: +49 (151) 14967823
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Felix Geisendörfer at Nov 28, 2014 at 12:14 pm ⇧
    The project is still going, but proprietary. There was a plan to open source parts of it, but I don’t know the current status of it since I’m no longer working on that project.

    Cheers
    Felix
    On 28.11.2014, at 08:10, exa.newspace@gmail.com wrote:

    Is this project still ongoing Felix ? I was searching for XMPP server in Go and did not find anything but that discussion...

    Le vendredi 27 décembre 2013 15:18:44 UTC+4, Felix Geisendoerfer a écrit :

    On Wed, Dec 25, 2013 at 11:46 PM, beevik wrote:
    I just updated the etree package at https://github.com/beevik/etree to support namespaces. (The change shouldn't break any existing code.)

    Excellent! I've now moved all our XML processing into a single package, so I'll make an attempt to replace gokogiri soon. Your library looks very promising, so I'll let you know how it goes / send patches if I come across problems!

    Cheers,
    --
    Felix Geisendörfer (felixge.de)
    Co-Founder, Debuggable Limited (debuggable.com)
    Mobile: +49 (151) 14967823
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Lor fal at Dec 14, 2013 at 11:51 am ⇧
    I have spent some time on how to implement XMPP in GO. I read the source
    code of jabberd2 and found that they use Expat to parse streaming xml.
    Currently I am working on a GO wrapper for Expat XML lib. I think Expat XML
    is very enough for parsing streaming XML. For construction of XML a normal
    DOM like lib is enough. Hope this can help you.
    On Friday, December 13, 2013 6:40:52 PM UTC+8, Felix Geisendoerfer wrote:

    Hi,

    I work on a large Go project implementing a XMPP [1] chat server. One of
    our biggest problem is handling XML in a clean way [3]. I'm currently
    trying to improve our approach to the problem, and writing this message to
    seek advise from the community. Some of this may seem negative, but I'm
    genuinely happy with Go overall, and also interested in contributing to
    making things better.

    The Marshal/Unmarshal interface of encoding/xml is at fundamental odds
    with the XMPP protocol. Unmarshal/Marshal expect XML that can be mapped to
    a static structure [2]. XMPP requires processing of XML that may have a
    static structure at its core, but can be extended with arbitrary elements
    and attributes by servers and clients alike [10].

    So for all practical purposes, XMPP requires a DOM [9] / Tree kind of
    data structure. Additionally XPath is really nice to have (but one can do
    without).

    And this is where things get tricky. There currently doesn't seem to be a
    mature DOM implementation for Go. xmldom-go [4] comes close, but has seen
    no commits for 9 months and looking at the code, I suspect it would take a
    lot of effort to finish it and clean it up. So at this point the only
    mature solution seems to be libxml2 via gokogiri [5], which is what we
    ended up with so far. However it has two major problems:

    1. It introduces manual memory management into a garbage collected
    environment. While Go makes it easy to create C bindings, they're
    incredibly annoying to use in practice because of this. SetFinalizer looks
    like it could help, but the guarantees it provides are currently too weak,
    and it's my understanding that there are no plans to change that.
    2. libxml2 has a really nice API for parsing streaming XML, but it doesn't
    really work [6] and I've had no luck getting the maintainers attention for
    the patch I provided [7].

    The end result of these two problems is that we're forced to do manual
    memory management (which has lead to bugs causing segfaults, panics,
    infinite loops and data corruption) as well as using encoding/xml to buffer
    up individual XML stanzas before parsing them for a second time with
    gokogiri : (.

    Additionally we've also been unable to use encoding/xml in more simplistic
    marshal scenarios involving namespaces, but maybe this is just our
    inability to use the API correctly [8]?

    Anyway, at this point we're between a rock and a hard place when it comes
    to handling xml. It seems like implementing a simple DOM pkg + support for
    basic XPath for our internal needs may be the best solution for now, but
    that's not going to make things any better for the community, as we'd
    rather not release another incomplete/ad-hoc solution into the ecosystem.

    So I'm wondering if anybody has advise. Are there plans of adding a
    DOM-style API to the go core? If yes, I'd like to help. If no, does anybody
    see any feasible alternatives for our situation? [11]

    Many thanks in advance,
    Felix Geisendörfer

    [1]: http://xmpp.org/xmpp-protocols/
    [2]: I use the term "static structure" to describe compile-time defined
    structures made up of structs and slices, rather than a Tree/DOM-like data
    structure.
    [3]: Have a drink if you laughed, otherwise continue on.
    [4]: http://godoc.org/bitbucket.org/rj/xmldom-go
    [5]: http://godoc.org/github.com/moovweb/gokogiri
    [6]: https://mail.gnome.org/archives/xml/2013-July/msg00015.html
    [7]: https://mail.gnome.org/archives/xml/2013-August/msg00025.html
    [8]: http://play.golang.org/p/RxnOZ1-I8E
    [9]: http://www.w3.org/TR/dom/
    [10]: The complexity of XMPP is worth a discussion as well, but we're
    stuck with it for this project.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Matt Ho at Dec 14, 2013 at 5:27 pm ⇧
    Would an option be to introduce a streaming filter that converts XML to json? You'd be able to leverage more of go's standard libraries at the cost of some up front fiddling.

    M

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Paddy Foran at Dec 24, 2013 at 4:10 pm ⇧
    If you decide to go the JSON route, a good starting point may
    be https://github.com/legastero/stanza.io
    On Saturday, December 14, 2013 12:26:56 PM UTC-5, Matt Ho wrote:

    Would an option be to introduce a streaming filter that converts XML to
    json? You'd be able to leverage more of go's standard libraries at the
    cost of some up front fiddling.

    M
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Konstantin Khomoutov at Dec 16, 2013 at 12:42 pm ⇧

    On Sat, 14 Dec 2013 03:51:51 -0800 (PST) lor fal wrote:

    I have spent some time on how to implement XMPP in GO. I read the
    source code of jabberd2 and found that they use Expat to parse
    streaming xml. Currently I am working on a GO wrapper for Expat XML
    lib. I think Expat XML is very enough for parsing streaming XML. For
    construction of XML a normal DOM like lib is enough. Hope this can
    help you.
    Wrapping a C library for parsing a stream of data coming from the
    network essentially means binding each parsing goroutine to a physical
    OS thread because AFAIK upon entry to a C code, the runtime will
    implicitly lock this goroutine to that physical OS thread the call has
    happened to be done on. On a highly-concurrent server with lots of
    clients this might result in suboptimal performance IMHO because the
    runtime will allocate lots of OS threads. See [1] for an some insight
    and search the group for the "runtime.LockOSThread" in the message
    bodies.

    1. https://groups.google.com/d/msg/golang-nuts/TesRETRf4KM/pWZUcdp0Ew8J

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Islan Dberry at Dec 16, 2013 at 3:43 pm ⇧

    On Monday, December 16, 2013 4:42:49 AM UTC-8, Konstantin Khomoutov wrote:
    Wrapping a C library for parsing a stream of data coming from the
    network essentially means binding each parsing goroutine to a physical
    OS thread
    This is not an issue for expat. The application is responsible for reading
    the input and pushing the data into the expat parser. The expat parser is
    not on the stack when the application is blocked waiting for input.

    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Lor fal at Dec 14, 2013 at 11:51 am ⇧
    I have spent some time on how to implement XMPP in GO. I read the source
    code of jabberd2 and found that they use Expat to parse streaming xml.
    Currently I am working on a GO wrapper for Expat XML lib. I think Expat XML
    is very enough for parsing streaming XML. For construction of XML a normal
    DOM like lib is enough. Hope this can help you.
    On Friday, December 13, 2013 6:40:52 PM UTC+8, Felix Geisendoerfer wrote:

    Hi,

    I work on a large Go project implementing a XMPP [1] chat server. One of
    our biggest problem is handling XML in a clean way [3]. I'm currently
    trying to improve our approach to the problem, and writing this message to
    seek advise from the community. Some of this may seem negative, but I'm
    genuinely happy with Go overall, and also interested in contributing to
    making things better.

    The Marshal/Unmarshal interface of encoding/xml is at fundamental odds
    with the XMPP protocol. Unmarshal/Marshal expect XML that can be mapped to
    a static structure [2]. XMPP requires processing of XML that may have a
    static structure at its core, but can be extended with arbitrary elements
    and attributes by servers and clients alike [10].

    So for all practical purposes, XMPP requires a DOM [9] / Tree kind of
    data structure. Additionally XPath is really nice to have (but one can do
    without).

    And this is where things get tricky. There currently doesn't seem to be a
    mature DOM implementation for Go. xmldom-go [4] comes close, but has seen
    no commits for 9 months and looking at the code, I suspect it would take a
    lot of effort to finish it and clean it up. So at this point the only
    mature solution seems to be libxml2 via gokogiri [5], which is what we
    ended up with so far. However it has two major problems:

    1. It introduces manual memory management into a garbage collected
    environment. While Go makes it easy to create C bindings, they're
    incredibly annoying to use in practice because of this. SetFinalizer looks
    like it could help, but the guarantees it provides are currently too weak,
    and it's my understanding that there are no plans to change that.
    2. libxml2 has a really nice API for parsing streaming XML, but it doesn't
    really work [6] and I've had no luck getting the maintainers attention for
    the patch I provided [7].

    The end result of these two problems is that we're forced to do manual
    memory management (which has lead to bugs causing segfaults, panics,
    infinite loops and data corruption) as well as using encoding/xml to buffer
    up individual XML stanzas before parsing them for a second time with
    gokogiri : (.

    Additionally we've also been unable to use encoding/xml in more simplistic
    marshal scenarios involving namespaces, but maybe this is just our
    inability to use the API correctly [8]?

    Anyway, at this point we're between a rock and a hard place when it comes
    to handling xml. It seems like implementing a simple DOM pkg + support for
    basic XPath for our internal needs may be the best solution for now, but
    that's not going to make things any better for the community, as we'd
    rather not release another incomplete/ad-hoc solution into the ecosystem.

    So I'm wondering if anybody has advise. Are there plans of adding a
    DOM-style API to the go core? If yes, I'd like to help. If no, does anybody
    see any feasible alternatives for our situation? [11]

    Many thanks in advance,
    Felix Geisendörfer

    [1]: http://xmpp.org/xmpp-protocols/
    [2]: I use the term "static structure" to describe compile-time defined
    structures made up of structs and slices, rather than a Tree/DOM-like data
    structure.
    [3]: Have a drink if you laughed, otherwise continue on.
    [4]: http://godoc.org/bitbucket.org/rj/xmldom-go
    [5]: http://godoc.org/github.com/moovweb/gokogiri
    [6]: https://mail.gnome.org/archives/xml/2013-July/msg00015.html
    [7]: https://mail.gnome.org/archives/xml/2013-August/msg00025.html
    [8]: http://play.golang.org/p/RxnOZ1-I8E
    [9]: http://www.w3.org/TR/dom/
    [10]: The complexity of XMPP is worth a discussion as well, but we're
    stuck with it for this project.
    --
    You received this message because you are subscribed to the Google Groups "golang-nuts" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgolang-nuts @
categoriesgo
postedDec 13, '13 at 10:40a
activeNov 28, '14 at 3:42p
posts21
users16
websitegolang.org

16 users in discussion

Felix Geisendoerfer: 3 posts Felix Geisendörfer: 3 posts Lor fal: 2 posts Will Norris: 1 post Codeprojpub: 1 post C Banning: 1 post Mateusz Czapliński: 1 post Gary b: 1 post Wkharold: 1 post Islan Dberry: 1 post Caleb Doxsey: 1 post Exa Newspace: 1 post Konstantin Khomoutov: 1 post Paddy Foran: 1 post Beevik: 1 post Matt Ho: 1 post

Content

People

Support

Translate

site design / logo © 2022 Grokbase