FAQ
I've got an established client-server application here where there
is now a need to shovel huge amounts of data (structured as lists of
lists) between the two, and the performance bottleneck has become
the amount of time spent parsing XML (it's taking 100% CPU on one or
other end of the connection and accounting for well over 50% of the
total call time, to the extent that it's having a greater impact on
performance than user interaction). And this is using SGMLOP for
parsing -- I dread to think what it would be like with a slower
parser. Anyone (Fredrik?) got any good ideas for tackling this
problem?

--
\S -- siona at chiark.greenend.org.uk -- http://www.chaos.org.uk/~sion/
___ | "Frankly I have no feelings towards penguins one way or the other"
\X/ | -- Arthur C. Clarke
her nu become? se bera eadward ofdun hl?ddre heafdes b?ce bump bump bump

Search Discussions

  • Diez B. Roggisch at Mar 21, 2006 at 4:59 pm

    Sion Arrowsmith wrote:

    I've got an established client-server application here where there
    is now a need to shovel huge amounts of data (structured as lists of
    lists) between the two, and the performance bottleneck has become
    the amount of time spent parsing XML (it's taking 100% CPU on one or
    other end of the connection and accounting for well over 50% of the
    total call time, to the extent that it's having a greater impact on
    performance than user interaction). And this is using SGMLOP for
    parsing -- I dread to think what it would be like with a slower
    parser. Anyone (Fredrik?) got any good ideas for tackling this
    problem?
    CORBA. Or TCP/IP directly. I know that isn't really the answer you're
    looking for - but there is an overhead parsing XML, and that can't go away.
    From what I read at SGMLOP's site I doub't that you can get much faster -
    because speed comes with sacrificing standard compliancy, which it already
    seems to do.

    Regards,

    Diez
  • Gregory Piñero at Mar 21, 2006 at 5:07 pm
    Perhaps there's a hardware solution? Could you run more servers in
    parallel? Or have some sort of load distribution system?

    (Just some generic server tips, I don't know much about XML-RPC)

    -Greg

    On 3/21/06, Diez B. Roggisch wrote:
    Sion Arrowsmith wrote:
    I've got an established client-server application here where there
    is now a need to shovel huge amounts of data (structured as lists of
    lists) between the two, and the performance bottleneck has become
    the amount of time spent parsing XML (it's taking 100% CPU on one or
    other end of the connection and accounting for well over 50% of the
    total call time, to the extent that it's having a greater impact on
    performance than user interaction). And this is using SGMLOP for
    parsing -- I dread to think what it would be like with a slower
    parser. Anyone (Fredrik?) got any good ideas for tackling this
    problem?
    CORBA. Or TCP/IP directly. I know that isn't really the answer you're
    looking for - but there is an overhead parsing XML, and that can't go away.
    From what I read at SGMLOP's site I doub't that you can get much faster -
    because speed comes with sacrificing standard compliancy, which it already
    seems to do.

    Regards,

    Diez
    --
    http://mail.python.org/mailman/listinfo/python-list

    --
    Gregory Pi?ero
    Chief Innovation Officer
    Blended Technologies
    (www.blendedtechnologies.com)
  • Paul McGuire at Mar 21, 2006 at 5:50 pm
    "Diez B. Roggisch" <deets at nospam.web.de> wrote in message
    news:48apnrFj7ambU1 at uni-berlin.de...
    Sion Arrowsmith wrote:
    I've got an established client-server application here where there
    is now a need to shovel huge amounts of data (structured as lists of
    lists) between the two, and the performance bottleneck has become
    the amount of time spent parsing XML (it's taking 100% CPU on one or
    other end of the connection and accounting for well over 50% of the
    total call time, to the extent that it's having a greater impact on
    performance than user interaction). And this is using SGMLOP for
    parsing -- I dread to think what it would be like with a slower
    parser. Anyone (Fredrik?) got any good ideas for tackling this
    problem?
    CORBA. Or TCP/IP directly.
    <snip>

    Beware, CORBA marshaling/unmarshaling can be similarly deadly, especially if
    you are passing Any's and have to marshal complex type codes. But if you
    are just sending arrays of octets, or strings of chars, then CORBA
    marshal/unmarshal will be quite fast.

    Of course, something will have to convert that array to meaningful data
    *somewhere*! You can't really eliminate software complexity, you can only
    move it around. (not original, I'm quoting Chris Stone, founder of the OMG.
    Also from Chris Stone - "I finally came up with a good definition for
    middleware. Middleware is the software nobody wants to pay for.")

    -- Paul
  • Diez B. Roggisch at Mar 21, 2006 at 9:32 pm

    Beware, CORBA marshaling/unmarshaling can be similarly deadly, especially if
    you are passing Any's and have to marshal complex type codes. But if you
    are just sending arrays of octets, or strings of chars, then CORBA
    marshal/unmarshal will be quite fast.
    Sure, any is no good at all. But I guess having a well-defined struct
    makes send corba the data comparably packed and (give or take endianess)
    directly in a representation close to what the result will look like. In
    contrast to XML, that wraps _everything_ in a plethora of bytes just for
    the fun of it...

    Of course, something will have to convert that array to meaningful data
    *somewhere*! You can't really eliminate software complexity, you can only
    move it around. (not original, I'm quoting Chris Stone, founder of the OMG.
    Also from Chris Stone - "I finally came up with a good definition for
    middleware. Middleware is the software nobody wants to pay for.")
    Also true - but you can make the matter of reinterpreting a bunch of
    bytes a mere cast (TCP/IP solution) or invoke a few thousand lines of
    code called XML-Parser. And on top of _that_ the XMLRPC itself. CORBA is
    certainly between these two, but on a scale of 5-20 times faster. Which
    might be the difference between a loaded but responsive server and a
    self-inflicted DDOS :)

    Diez
  • Skip at Mar 21, 2006 at 5:55 pm

    Diez> Sion Arrowsmith wrote:
    I've got an established client-server application here where there
    is now a need to shovel huge amounts of data (structured as lists of
    lists) between the two, and the performance bottleneck has become
    the amount of time spent parsing XML ...
    Diez> CORBA. Or TCP/IP directly. I know that isn't really the answer
    Diez> you're looking for - but there is an overhead parsing XML, and
    Diez> that can't go away.

    Ah, but if both ends of the pipe happen to be running Python, you can
    cheat. ;-) Run your list through marshal.dumps() before passing it to
    xmlrpclib, then undo the marshalling on the other end.
    biglist = ["abc"] * 50
    biglist = [biglist] * 50
    import xmlrpclib
    import marshal
    x = xmlrpclib.dumps((biglist,))
    len(x)
    92331
    x.count("<")
    10310
    y = xmlrpclib.dumps((marshal.dumps(biglist),))
    len(y)
    20324
    y.count("<")
    8

    If you can swing it, I'd be willing to bet you a beer your XML parsing time
    will all but disappear.

    If you're not using xmlrpclib with some sort of C helper (like sgmlop), try
    that as well. Big difference parsing XML in C and Python.

    Skip
  • Sion Arrowsmith at Mar 23, 2006 at 1:52 pm

    wrote:
    Diez> Sion Arrowsmith wrote:
    I've got an established client-server application here where there
    is now a need to shovel huge amounts of data (structured as lists of
    lists) between the two, and the performance bottleneck has become
    the amount of time spent parsing XML ...
    Diez> CORBA. Or TCP/IP directly. I know that isn't really the answer
    Diez> you're looking for - but there is an overhead parsing XML, and
    Diez> that can't go away.
    Ah, but if both ends of the pipe happen to be running Python, you can
    cheat. ;-) Run your list through marshal.dumps() before passing it to
    xmlrpclib, then undo the marshalling on the other end.
    [ ... ]
    If you can swing it, I'd be willing to bet you a beer your XML parsing time
    will all but disappear.
    I wouldn't take you up on that bet, because I'd already considered
    that as a likely solution, and while I was asking here another member
    of the team went away and implemented it for me. Except with cPickle
    (as suggested elsewhere in the thread) -- we've got DateTime objects
    in there, which marshal chokes on. Initial tests show you'd've
    comfortably won your beer: 16s XML parsing down to 0.5s parsing plus
    1.5s unpickling.
    If you're not using xmlrpclib with some sort of C helper (like sgmlop), try
    that as well. Big difference parsing XML in C and Python.
    Diez edited out the bit where I said we were using sgmlop and worried
    about what it would be like if we weren't 8-)

    --
    \S -- siona at chiark.greenend.org.uk -- http://www.chaos.org.uk/~sion/
    ___ | "Frankly I have no feelings towards penguins one way or the other"
    \X/ | -- Arthur C. Clarke
    her nu become? se bera eadward ofdun hl?ddre heafdes b?ce bump bump bump
  • Rene Pijlman at Mar 21, 2006 at 5:28 pm
    Sion Arrowsmith:
    I've got an established client-server application here where there
    is now a need to shovel huge amounts of data (structured as lists of
    lists) between the two, and the performance bottleneck has become
    the amount of time spent parsing XML
    http://xmlsucks.org/
    Anyone got any good ideas for tackling this problem?
    cPickle.
    http://docs.python.org/lib/module-cPickle.html

    --
    Ren? Pijlman

    Wat wil jij leren? http://www.leren.nl
  • Fredrik Lundh at Mar 21, 2006 at 5:36 pm

    Sion Arrowsmith wrote:

    I've got an established client-server application here where there
    is now a need to shovel huge amounts of data (structured as lists of
    lists) between the two, and the performance bottleneck has become
    the amount of time spent parsing XML (it's taking 100% CPU on one or
    other end of the connection and accounting for well over 50% of the
    total call time, to the extent that it's having a greater impact on
    performance than user interaction). And this is using SGMLOP for
    parsing -- I dread to think what it would be like with a slower
    parser. Anyone (Fredrik?) got any good ideas for tackling this
    problem?
    the cElementTree unmarshaller on this page

    http://effbot.org/zone/element-iterparse.htm#incremental-decoding

    is a bit faster than the one in xmlrpclib. (to get more performance,
    you probably need a library that allows you to register decoders on
    the C level.)

    </F>
  • Gregarican at Mar 21, 2006 at 5:38 pm

    Sion Arrowsmith wrote:

    shovel huge amounts of data
    That right there basically takes XML-RPC off the table as a viable
    solution. No matter how much fine tuning you try large amounts of data
    don't bode well using XML-RPC. The other posted alternatives are
    certainly worth a shot. From CORBA to cPickle to even a basic TCP
    socket connection would be more efficient. I have used XML-RPC for very
    small recordset transfers but I wouldn't use it for anything large.
  • Steve Holden at Mar 21, 2006 at 6:16 pm

    gregarican wrote:
    Sion Arrowsmith wrote:

    shovel huge amounts of data

    That right there basically takes XML-RPC off the table as a viable
    solution. No matter how much fine tuning you try large amounts of data
    don't bode well using XML-RPC. The other posted alternatives are
    certainly worth a shot. From CORBA to cPickle to even a basic TCP
    socket connection would be more efficient. I have used XML-RPC for very
    small recordset transfers but I wouldn't use it for anything large.
    <rant>
    I am continually surprised that the industry as a whole has become so
    enamoured of XML that people persist in using it for tasks it was never
    designed nor intended to be used for.
    </rant>

    I suppose there *was* a good reason for using XML-RPC in the first place?

    regards
    Steve
    --
    Steve Holden +44 150 684 7255 +1 800 494 3119
    Holden Web LLC/Ltd www.holdenweb.com
    Love me, love my blog holdenweb.blogspot.com
  • Skip at Mar 21, 2006 at 6:25 pm
    Steve> I suppose there *was* a good reason for using XML-RPC in the
    Steve> first place?

    I don't know about the OP, but in my case it was a drop-dead simple
    cross-language RPC protocol. At one point Mojam's XML-RPC server talked to
    clients written in Python, Perl and Objective-C (via a Java bridge, no
    less).

    Skip
  • Gregarican at Mar 21, 2006 at 6:46 pm

    Skip wrote:

    I don't know about the OP, but in my case it was a drop-dead simple
    cross-language RPC protocol.
    Exactly. RPC implementations. Typically these are a quick "do this with
    this" sent from the client, with a brief "okay, I did that, here's the
    result" back from the server. Shovelling huge amounts of data using any
    XML encapsulation leads to a certain code smell at the very least.
  • Steve Holden at Mar 21, 2006 at 8:19 pm

    gregarican wrote:
    Skip wrote:

    I don't know about the OP, but in my case it was a drop-dead simple
    cross-language RPC protocol.

    Exactly. RPC implementations. Typically these are a quick "do this with
    this" sent from the client, with a brief "okay, I did that, here's the
    result" back from the server. Shovelling huge amounts of data using any
    XML encapsulation leads to a certain code smell at the very least.
    Especially when people try to pass large amounts of free text (like
    Python scripts) as attribute values ;-)

    regards
    Steve
    --
    Steve Holden +44 150 684 7255 +1 800 494 3119
    Holden Web LLC/Ltd www.holdenweb.com
    Love me, love my blog holdenweb.blogspot.com
  • Sion Arrowsmith at Mar 23, 2006 at 2:29 pm

    wrote:
    Steve> I suppose there *was* a good reason for using XML-RPC in the
    Steve> first place?
    I don't know about the OP, but in my case it was a drop-dead simple
    cross-language RPC protocol.
    I am the OP and *I* don't know if there was a good reason for using
    XML-RPC in the first place. It's someone else's code, and they're
    no longer with the company. I can see it being justifiable at the
    time: (a) single developer writing both server and client doesn't
    need to think about the implemention of their communication (b) in
    the future there may be other clients in other languages (as above)
    and (c) up until recently, the volume of data being passed back and
    forth wasn't high enough for XML parsing performance to be of much
    significance. I've known XML parsing makes XML-RPC suck since, er,
    before XML-RPC was invented. (At about the same time that SOAP was
    being developed, we developed a prototype component system using
    XML for message passing, then threw it away when it was clear that
    the XML parsing was a major bottleneck.)

    --
    \S -- siona at chiark.greenend.org.uk -- http://www.chaos.org.uk/~sion/
    ___ | "Frankly I have no feelings towards penguins one way or the other"
    \X/ | -- Arthur C. Clarke
    her nu become? se bera eadward ofdun hl?ddre heafdes b?ce bump bump bump
  • Irmen de Jong at Mar 22, 2006 at 1:42 am

    Sion Arrowsmith wrote:
    I've got an established client-server application here where there
    is now a need to shovel huge amounts of data (structured as lists of
    lists) between the two, and the performance bottleneck has become
    the amount of time spent parsing XML
    Any chance of replacing the RPC protocol by something else
    that is more efficient? If so, and you're in a 100%-python situation,
    have a look at Pyro: http://pyro.sourceforge.net

    (Pyro basically uses "native" Python pickling as marshaling)

    --Irmen

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedMar 21, '06 at 4:31p
activeMar 23, '06 at 2:29p
posts16
users10
websitepython.org

People

Translate

site design / logo © 2022 Grokbase