Hi

One of the OpenAMQ team kindly alerted me to this: https://launchpad.net/txamqp

"Python library for communicating with AMQP peers and brokers using
Twisted. This project contains all the necessary code to connect,
send and receive messages to/from an AMQP-compliant peer or broker
(Qpid, OpenAMQ, RabbitMQ) using Twisted."

Is anyone using this, or using Twisted?

Let us know - I'd be delighted to understand the use cases better.

alexis

Search Discussions

  • Holger Hoffstätte at Aug 21, 2008 at 4:43 pm

    Alexis Richardson wrote:
    One of the OpenAMQ team kindly alerted me to this: https://launchpad.net/txamqp kewl!
    Is anyone using this, or using Twisted?

    Let us know - I'd be delighted to understand the use cases better.
    You mean use cases for Twisted? It's a very nifty Python framework (in the
    good sense) with many transports for easily building nonblocking/async
    clients & servers, kind of like Java's NIO. This is more important for
    python than Java because of its threading.
    As far as AMQP is concerned I planned to play with Twisted (while trying
    to improve my python at the same time) and had thought about basing a
    python AMQP client on Twisted, but with the shared connection for multiple
    channels it turned out to be not so useful after all. The easy framework
    integration might still be very valuable though.

    As an aside, does anybody know whether the AMQP spec mandates the m:1
    channel:connection relationship, or is that just what's implemented right
    now in Rabbit? IMHO clients in general could also benefit greatly from
    bonding multiple connections together; the Java client might be able to
    use NIO for that. This could cut down connection contention with many
    channels and allow for better network utilization, though it would need to
    be handled by the broker as well. Just curious.

    Holger
  • Matthias Radestock at Aug 23, 2008 at 8:23 am
    Holger,

    Holger Hoffst?tte wrote:
    As an aside, does anybody know whether the AMQP spec mandates the m:1
    channel:connection relationship, or is that just what's implemented right
    now in Rabbit?
    The spec allows for multiple channels per connection, and multiple
    connections per client.
    IMHO clients in general could also benefit greatly from
    bonding multiple connections together; the Java client might be able to
    use NIO for that. This could cut down connection contention with many
    channels and allow for better network utilization, though it would need to
    be handled by the broker as well. Just curious.
    Are you referring to ethernet bonding? Please elaborate.


    Matthias.
  • Holger Hoffstätte at Aug 24, 2008 at 5:29 pm

    Matthias Radestock wrote:
    Holger Hoffst?tte wrote:
    As an aside, does anybody know whether the AMQP spec mandates the m:1
    channel:connection relationship, or is that just what's implemented right
    now in Rabbit?
    The spec allows for multiple channels per connection, and multiple
    connections per client.
    I'm afraid your answer is completely correct and yet not what I wanted to
    know. ;-)
    I'll admit the question was ambiguous.
    IMHO clients in general could also benefit greatly from
    bonding multiple connections together; the Java client might be able to
    use NIO for that. This could cut down connection contention with many
    channels and allow for better network utilization, though it would
    need to
    be handled by the broker as well. Just curious.
    Are you referring to ethernet bonding? Please elaborate.
    More like "socket bonding" if that makes sense - the question (and
    half-baked idea) was whether a logical connection could be implemented in
    terms of multiple sockets (when using TCP), without the client app doing
    this manually/explicitly. Allowing multiple sockets could prevent channel
    starvation and therefore also better QoS schemes, but on second thought it
    is not clear whether just delegating to the OS is such a good idea after all.

    A typical scenario would be a couple of channels that send/receive mostly
    small messages at varying rates but with low latency requirement, yet the
    one oddball sends huge messages but doesn't care about the transfer speed.
    Without some kind of fairness on the framing level (e.g. by dividing up
    the one shared socket among all channels with some kind of quota, channel
    priority or deadline), the one big sender could starve the other small
    messages.

    I'm not sure if this is important at this point, or even whether doing
    this kind of QoS on the frame level (ala a packet scheduler) or on a
    per-socket level would be better. I dind't find any useful QoS information
    in the spec except for the note on the Channel class and the per-message
    credit-based "flow control" concept, but to me that seems of little use
    for starvation control on the transport level. From what I can tell that
    (and session ordering) is what the frame "Tracks" in 0.10 are about, I
    didn't see that in 0.8/0.9.

    Enough elaborated, I guess. :)

    regards
    Holger
  • Matthias Radestock at Aug 24, 2008 at 6:29 pm
    Holger,

    Holger Hoffst?tte wrote:
    Matthias Radestock wrote:
    Are you referring to ethernet bonding? Please elaborate.
    More like "socket bonding" if that makes sense - the question (and
    half-baked idea) was whether a logical connection could be implemented in
    terms of multiple sockets (when using TCP), without the client app doing
    this manually/explicitly. Allowing multiple sockets could prevent channel
    starvation and therefore also better QoS schemes, but on second thought it
    is not clear whether just delegating to the OS is such a good idea after all.
    What's wrong with having multiple logical (and thus TCP) connections?
    Does your application depend on having a single logical connection? I'd
    find that surprising given that the connection doesn't feature as a
    logical entity in most AMQP operations; most things happen at the
    granularity of channels.
    A typical scenario would be a couple of channels that send/receive mostly
    small messages at varying rates but with low latency requirement, yet the
    one oddball sends huge messages but doesn't care about the transfer speed.
    Without some kind of fairness on the framing level (e.g. by dividing up
    the one shared socket among all channels with some kind of quota, channel
    priority or deadline), the one big sender could starve the other small
    messages.
    The broker does a reasonably good job of treating channels fairly. Large
    messages will be split across multiple frames, allowing smaller messages
    from other channels to be interleaved.

    As you say, sending the messages across different TCP connections just
    delegates the problem to the OS. That's probably a good thing, but
    whether it actually gives you any better fairness only testing will reveal.


    Matthias.
  • Holger Hoffstätte at Aug 24, 2008 at 7:19 pm

    Matthias Radestock wrote:
    What's wrong with having multiple logical (and thus TCP) connections?
    Nothing, except that each app would have to write this over and over
    again, so I was wondering if something like a MultiSocketConnectionFactory
    could return a connection that uses multiple sockets under the hood (I'm
    looking at this purely with an eye on the Java client since that is what I
    use for experiments). Then channels could be "bound" to one of the
    internal sockets depending on their "priority class" or whatever we call it.

    However after spending some time with the 0.10 spec this afternoon I think
    all this will be better solved on the framing level.
    Does your application depend on having a single logical connection? I'd
    No, I was just thinking out loud.
    The broker does a reasonably good job of treating channels fairly. Large
    messages will be split across multiple frames, allowing smaller messages
    from other channels to be interleaved.
    Yes, that dawned on me just as your mail arrived..probably because each
    channel has its own process and gets a fair share for writing. That solves
    the problem for the broker, which is good..but leaves the client libs,
    which is complicated but probably also not very important for now.

    thanks!
    Holger
  • Brian Granger at Aug 21, 2008 at 5:14 pm

    Hi

    One of the OpenAMQ team kindly alerted me to this: https://launchpad.net/txamqp

    "Python library for communicating with AMQP peers and brokers using
    Twisted. This project contains all the necessary code to connect,
    send and receive messages to/from an AMQP-compliant peer or broker
    (Qpid, OpenAMQ, RabbitMQ) using Twisted."

    Is anyone using this, or using Twisted?
    I have used Twisted extensively (an asynchronous networking library
    for Python), and saw the announcement for txamqp, but I have not
    looked at it. But, I do think that having a really good Twisted+AMQP
    library would make AMQP much more usable from python.
    Let us know - I'd be delighted to understand the use cases better.
    I think the most important point about Twisted is that is has become
    _the way_ in Python for handling asynchronous networking. From this
    standpoint, I would consider a Twisted+AMQP library to be to most
    natural approach for having Python bindings to AMQP.

    I have used the Qpid Python bindings and while they work OK, I found
    them lacking in how asynchronous things are handled.

    I am more than willing to discuss this more if people want.

    Brian
    alexis

    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Drew Smathers at Aug 21, 2008 at 5:57 pm

    On Thu, Aug 21, 2008 at 1:14 PM, Brian Granger wrote:
    I think the most important point about Twisted is that is has become
    _the way_ in Python for handling asynchronous networking. From this
    standpoint, I would consider a Twisted+AMQP library to be to most
    natural approach for having Python bindings to AMQP.

    I have used the Qpid Python bindings and while they work OK, I found
    them lacking in how asynchronous things are handled.

    I am more than willing to discuss this more if people want.
    Some interesting implications are that by using Twisted, some pretty
    useful adapters could be build between clients speaking protocols
    specific to Twisted or implemented in Twisted (AMP and experimental
    Protocol Buffers impl come to mind). I need to put
    evaluating/exercising txamqp on my TODO list as well. Having a robust
    twisted AMQP implementation would benefit both projects.


    --
    \\\\\/\"/\\\\\\\\\\\
    \\\\/ // //\/\\\\\\\
    \\\/ \\// /\ \/\\\\
    \\/ /\/ / /\/ /\ \\\
    \/ / /\/ /\ /\\\ \\
    / /\\\ /\\\ \\\\\/\
    \/\\\\\/\\\\\/\\\\\\
    d.p.s
  • Paul C. Nendick at Aug 21, 2008 at 6:10 pm
    Afternoon from London All,

    Yes, there are those of us using RabbitMQ with Twisted. Actually, on
    my present project, we're using it very heavily - it's great - and
    we've made a proper, working twisted protocol implementation for AMQP.
    I'm working with my employer on open sourcing this bit of code but
    it's unlikely to happen soon. If and when it does, I'll release it as
    either a patch to txamqp or as part of this project:
    http://www.peloton-grid.net/

    Some observations about txamqp:
    * it's very alpha and clearly incomplete
    * it's fundamentally misunderstood the way AMQP routes of messages
    over separate channels. Deferred calls often don't fire since txamqp
    is waiting for the response on the wrong channel. Oops!
    * txamqp is multi-threaded. Tsk tsk, this is a no-no for twisted
    protocol handlers
    * QPID sucks. Badly, just look at the code.
    * QPID's autogeneration from the XML spec however is very cool
    * the txamqp unit tests are very good and quite easy to adapt to be
    made asynchronous

    We've resolved these and other issues. In the end, it will help me
    open source the code to know the level of your interest in it. I'll
    announce any news on this list.

    regards,

    /p

    PS: anyone else going to PyCon UK?
  • Brian Granger at Aug 21, 2008 at 6:51 pm

    Yes, there are those of us using RabbitMQ with Twisted. Actually, on
    my present project, we're using it very heavily - it's great - and
    we've made a proper, working twisted protocol implementation for AMQP.
    I'm working with my employer on open sourcing this bit of code but
    it's unlikely to happen soon. If and when it does, I'll release it as
    either a patch to txamqp or as part of this project:
    http://www.peloton-grid.net/
    Fantastic.
    Some observations about txamqp:
    * it's very alpha and clearly incomplete
    * it's fundamentally misunderstood the way AMQP routes of messages
    over separate channels. Deferred calls often don't fire since txamqp
    is waiting for the response on the wrong channel. Oops!
    * txamqp is multi-threaded. Tsk tsk, this is a no-no for twisted
    protocol handlers
    * QPID sucks. Badly, just look at the code.
    * QPID's autogeneration from the XML spec however is very cool
    * the txamqp unit tests are very good and quite easy to adapt to be
    made asynchronous
    I haven't looked at txamqp enough to see these issues. But yes,
    because qpid uses threads, it won't mix well with Twisted.
    We've resolved these and other issues. In the end, it will help me
    open source the code to know the level of your interest in it. I'll
    announce any news on this list.
    I would be very interested in see a robust Twisted+AMQP library.

    Cheers,

    Brian
    regards,

    /p

    PS: anyone else going to PyCon UK?

    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Ben Hood at Aug 21, 2008 at 7:47 pm
    Paul
    On Thu, Aug 21, 2008 at 7:10 PM, Paul C. Nendick wrote:
    * txamqp is multi-threaded. Tsk tsk, this is a no-no for twisted
    protocol handlers
    You might then be interested in Barry Pedersen's library that doesn't thread:
    http://barryp.org/software/py-amqplib/
    * QPID's autogeneration from the XML spec however is very cool
    Barry's lib generates this from the XML spec as well, but does it
    statically. In fact, this is the approach that RabbitMQ takes, see the
    (Python-based) codegen module:
    http://hg.rabbitmq.com/rabbitmq-codegen/

    Also, several other libraries take this compile time approach to codec
    handling as well.

    HTH,

    Ben
  • Kyle at Aug 21, 2008 at 11:22 pm

    On Thu, Aug 21, 2008 at 3:47 PM, Ben Hood wrote:
    You might then be interested in Barry Pedersen's library that doesn't
    thread:
    http://barryp.org/software/py-amqplib/
    I've been trying to find a good solution to using amqp within a Twisted
    framework for a while now. Barry's library was the first thing we tried, but
    unfortunately found that it caused errors within the Twisted reactor that
    were very hard to track down, and using wait() in a loop doesn't work well
    when the rest of the code is asynchronous. Eventually we settled on using
    RabbitMQ's Java bindings through JPype <http://jpype.sourceforge.net/>,
    which has its own problems, not the least of which is bringing a JVM into
    the process, which I'd love to get rid of.
    We've resolved these and other issues. In the end, it will help me
    open source the code to know the level of your interest in it. I'll
    announce any news on this list.
    A solid Twisted AMQP library would be great, we've been thinking about
    working on one for a while now, but it got lost in the background after we
    ran with the Java client with no issues. Definitely a lot of interest here.

    Thanks,
    Kyle
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20080821/c8367534/attachment.htm
  • Alexis Richardson at Aug 22, 2008 at 8:56 am
    Kyle, Paul, Brian, et al.

    Thanks very much indeed for the rapid fire responses.
    On Fri, Aug 22, 2008 at 12:22 AM, Kyle wrote:

    We've resolved these and other issues. In the end, it will help me
    open source the code to know the level of your interest in it. I'll
    announce any news on this list.
    A solid Twisted AMQP library would be great, we've been thinking about
    working on one for a while now, but it got lost in the background after we
    ran with the Java client with no issues. Definitely a lot of interest here.
    I have as they say 'reached out to' Terry from Fluidinfo - the authors
    of txAMQP - to let him know there is interest.

    He's terrycojones on twitter.

    alexis
  • Rui Lopes at Aug 22, 2008 at 9:33 am

    Alexis Richardson wrote:
    We've resolved these and other issues. In the end, it will help me
    open source the code to know the level of your interest in it. I'll
    announce any news on this list.
    A solid Twisted AMQP library would be great, we've been thinking about
    working on one for a while now, but it got lost in the background after we
    ran with the Java client with no issues. Definitely a lot of interest here.
    I have as they say 'reached out to' Terry from Fluidinfo - the authors
    of txAMQP - to let him know there is interest.

    He's terrycojones on twitter.
    Just want to wave my hand!

    Thanks for reaching him out, and hope to see then release it :-)

    Best regards,
    Rui Lopes
  • Guillaume Theoret at Aug 22, 2008 at 4:24 pm

    On Thu, Aug 21, 2008 at 7:22 PM, Kyle wrote:
    On Thu, Aug 21, 2008 at 3:47 PM, Ben Hood wrote:

    You might then be interested in Barry Pedersen's library that doesn't
    thread:
    http://barryp.org/software/py-amqplib/
    I've been trying to find a good solution to using amqp within a Twisted
    framework for a while now. Barry's library was the first thing we tried, but
    unfortunately found that it caused errors within the Twisted reactor that
    were very hard to track down, and using wait() in a loop doesn't work well
    when the rest of the code is asynchronous. Eventually we settled on using
    RabbitMQ's Java bindings through JPype, which has its own problems, not the
    least of which is bringing a JVM into the process, which I'd love to get rid
    of.
    We've resolved these and other issues. In the end, it will help me
    open source the code to know the level of your interest in it. I'll
    announce any news on this list.
    A solid Twisted AMQP library would be great, we've been thinking about
    working on one for a while now, but it got lost in the background after we
    ran with the Java client with no issues. Definitely a lot of interest here.
    Same here. I tried for a while with Barry's library but couldn't
    figure out a couple of things (probably my fault though not his, still
    very new at AMQP and don't fully understand how a lot of the stuff is
    supposed to be done yet) and since I got the Java client going so well
    and easily we went with that.

    Here we (and I) definitely prefer coding in python =)
    Thanks,
    Kyle
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedAug 21, '08 at 3:32p
activeAug 24, '08 at 7:19p
posts15
users10
websiterabbitmq.com
irc#rabbitmq

People

Translate

site design / logo © 2022 Grokbase