Hello.

This note is longer than I thought it would be, but I do not believe
my situation is complicated.

I am a RabbitMQ newcomer and am trying to identify which exchange type
to use for my application.

I have a publisher P that needs to send messages about database table
updates. ?A message will bear the name of a single table name,
followed by some opaque application data:

Logically,

message == tableName | somedata

So consider a simplified database with three tables of interest: ?T1,
T2, and T3.

There are two consumers, C1 and C2.

C1 needs to receive all messages concerning tables T1 and T2.

C2 needs to receive all messages concerning tables T2 and T3.

By implication, C1 and C2 must both receive messages about T2, where
their interests overlap.

Ideally, I want to publish to a single exchange, and, therefore, my
consumers also bind to this same single exchange. ?I say this because
in fact there are a lot more than three tables to treat - there are
almost 200. ?A proliferation of exchanges per-table would be not good.

I am considering a topic or header exchange.

If a header exchange, I was thinking of the publisher P putting the
concerned table name in a "table header" and the consumers binding to
the exchange with an interest in receiving messages with a table
header value equal to T1, T2, or T3 (I believe x-match == any would be
appropriate when the consumer binds). ?But some of my readings on
header exhchanges here

http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2011-January/010935.html

suggest that header exchanges may not be as useful as maybe the name
suggests. Maybe I'm being overly paranoid.

And I consider topic exchanges because I know they admit messages
about T2 consumed by C1 still being available to C2.

Would someone be kind enough to suggest approaches to discern which
type of exchange to use.

Thank you very kindly.

--
Mark

Search Discussions

  • Mark Petrovic at Jan 10, 2012 at 10:20 pm
    Talking to myself here (I have no problem with that :-)), I think I
    have at least one solution to this.

    - Create a single exchange E
    - The Publisher P creates a channel with a single queue and routing
    key with the same names as the Table T in question.
    - Publisher publishes a message to channel
    - The Consumer C1 iterates over the table names T1 and T2 he cares
    about and binds his channel to queues and routing keys with the same
    name as the table name iterated over
    - The Consumer C1 then starts a thread for each of these queues, and
    whose run() method calls channel.basicConsume(queueName, autoAck,
    rabbitConsumer)

    where rabbitConsumer extends com.rabbitmq.client.DefaultConsumer.

    While this seems to work, and is a bit easier to understand how to
    program than header exchanges (if a header exchange would even work
    for me - I wonder), I dont' like the thread-per-queue I need to spawn
    to handle messages inbound for each queue.

    What would be nice is if I could call channel.basicConsume(autoAck,
    rabbitConsumer), which would read messages off all the queues that
    were bound to that channel.

    While I have something working, I'm not yet real thrilled with my
    programming model.

    Anybody?

    Thanks!
    On Tue, Jan 10, 2012 at 10:28 AM, Mark Petrovic wrote:
    Hello.

    This note is longer than I thought it would be, but I do not believe
    my situation is complicated.

    I am a RabbitMQ newcomer and am trying to identify which exchange type
    to use for my application.

    I have a publisher P that needs to send messages about database table
    updates. ?A message will bear the name of a single table name,
    followed by some opaque application data:

    Logically,

    message == tableName | somedata

    So consider a simplified database with three tables of interest: ?T1,
    T2, and T3.

    There are two consumers, C1 and C2.

    C1 needs to receive all messages concerning tables T1 and T2.

    C2 needs to receive all messages concerning tables T2 and T3.

    By implication, C1 and C2 must both receive messages about T2, where
    their interests overlap.

    Ideally, I want to publish to a single exchange, and, therefore, my
    consumers also bind to this same single exchange. ?I say this because
    in fact there are a lot more than three tables to treat - there are
    almost 200. ?A proliferation of exchanges per-table would be not good.

    I am considering a topic or header exchange.

    If a header exchange, I was thinking of the publisher P putting the
    concerned table name in a "table header" and the consumers binding to
    the exchange with an interest in receiving messages with a table
    header value equal to T1, T2, or T3 (I believe x-match == any would be
    appropriate when the consumer binds). ?But some of my readings on
    header exhchanges here

    http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2011-January/010935.html

    suggest that header exchanges may not be as useful as maybe the name
    suggests. ?Maybe I'm being overly paranoid.

    And I consider topic exchanges because I know they admit messages
    about T2 consumed by C1 still being available to C2.

    Would someone be kind enough to suggest approaches to discern which
    type of exchange to use.

    Thank you very kindly.

    --
    Mark


    --
    Mark
  • Mark Petrovic at Jan 11, 2012 at 12:46 am
    I think I have a solution, better than one based on a header exchange
    or the scheme I outline below.

    It's basically this solution:

    http://www.rabbitmq.com/tutorials/tutorial-five-java.html

    where the routing keys are the table names. This gives me a single
    consumer queue bound to mulitple routing keys, which gets my job done.


    On Tue, Jan 10, 2012 at 2:20 PM, Mark Petrovic wrote:
    Talking to myself here (I have no problem with that :-)), I think I
    have at least one solution to this.

    - Create a single exchange E
    - The Publisher P creates a channel with a single queue and routing
    key with the same names as the Table T in question.
    - Publisher publishes a message to channel
    - The Consumer C1 iterates over the table names T1 and T2 he cares
    about and binds his channel to queues and routing keys with the same
    name as the table name iterated over
    - The Consumer C1 then starts a thread for each of these queues, and
    whose run() method calls channel.basicConsume(queueName, autoAck,
    rabbitConsumer)

    where rabbitConsumer extends com.rabbitmq.client.DefaultConsumer.

    While this seems to work, and is a bit easier to understand how to
    program than header exchanges (if a header exchange would even work
    for me - I wonder), I dont' like the thread-per-queue I need to spawn
    to handle messages inbound for each queue.

    What would be nice is if I could call channel.basicConsume(autoAck,
    rabbitConsumer), which would read messages off all the queues that
    were bound to that channel.

    While I have something working, I'm not yet real thrilled with my
    programming model.

    Anybody?

    Thanks!
    On Tue, Jan 10, 2012 at 10:28 AM, Mark Petrovic wrote:
    Hello.

    This note is longer than I thought it would be, but I do not believe
    my situation is complicated.

    I am a RabbitMQ newcomer and am trying to identify which exchange type
    to use for my application.

    I have a publisher P that needs to send messages about database table
    updates. ?A message will bear the name of a single table name,
    followed by some opaque application data:

    Logically,

    message == tableName | somedata

    So consider a simplified database with three tables of interest: ?T1,
    T2, and T3.

    There are two consumers, C1 and C2.

    C1 needs to receive all messages concerning tables T1 and T2.

    C2 needs to receive all messages concerning tables T2 and T3.

    By implication, C1 and C2 must both receive messages about T2, where
    their interests overlap.

    Ideally, I want to publish to a single exchange, and, therefore, my
    consumers also bind to this same single exchange. ?I say this because
    in fact there are a lot more than three tables to treat - there are
    almost 200. ?A proliferation of exchanges per-table would be not good.

    I am considering a topic or header exchange.

    If a header exchange, I was thinking of the publisher P putting the
    concerned table name in a "table header" and the consumers binding to
    the exchange with an interest in receiving messages with a table
    header value equal to T1, T2, or T3 (I believe x-match == any would be
    appropriate when the consumer binds). ?But some of my readings on
    header exhchanges here

    http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2011-January/010935.html

    suggest that header exchanges may not be as useful as maybe the name
    suggests. ?Maybe I'm being overly paranoid.

    And I consider topic exchanges because I know they admit messages
    about T2 consumed by C1 still being available to C2.

    Would someone be kind enough to suggest approaches to discern which
    type of exchange to use.

    Thank you very kindly.

    --
    Mark


    --
    Mark


    --
    Mark
  • Alvaro Videla at Jan 11, 2012 at 9:28 am
    Hi Mark,

    Yes a topic exchange can be ideal when you need to "listen" based on
    several patterns. Also this exchange leaves the possibility open for new
    table names,
    You just have to think about the binding pattern that you want to use.

    Also keep in mind that if you want message X to arrive at consumer A and B
    then you need one separate queue per consumer bound to that exchange using
    a binding key that matches the routing key used when publishing message X.
    What this means is that messages are "fanout'ed" at the exchange, but once
    they arrive at a queue then messages are pulled out individually and once
    gone from the queue is not seen again.

    Cheers,

    Alvaro
    On Wed, Jan 11, 2012 at 1:46 AM, Mark Petrovic wrote:

    I think I have a solution, better than one based on a header exchange
    or the scheme I outline below.

    It's basically this solution:

    http://www.rabbitmq.com/tutorials/tutorial-five-java.html

    where the routing keys are the table names. This gives me a single
    consumer queue bound to mulitple routing keys, which gets my job done.


    On Tue, Jan 10, 2012 at 2:20 PM, Mark Petrovic wrote:
    Talking to myself here (I have no problem with that :-)), I think I
    have at least one solution to this.

    - Create a single exchange E
    - The Publisher P creates a channel with a single queue and routing
    key with the same names as the Table T in question.
    - Publisher publishes a message to channel
    - The Consumer C1 iterates over the table names T1 and T2 he cares
    about and binds his channel to queues and routing keys with the same
    name as the table name iterated over
    - The Consumer C1 then starts a thread for each of these queues, and
    whose run() method calls channel.basicConsume(queueName, autoAck,
    rabbitConsumer)

    where rabbitConsumer extends com.rabbitmq.client.DefaultConsumer.

    While this seems to work, and is a bit easier to understand how to
    program than header exchanges (if a header exchange would even work
    for me - I wonder), I dont' like the thread-per-queue I need to spawn
    to handle messages inbound for each queue.

    What would be nice is if I could call channel.basicConsume(autoAck,
    rabbitConsumer), which would read messages off all the queues that
    were bound to that channel.

    While I have something working, I'm not yet real thrilled with my
    programming model.

    Anybody?

    Thanks!
    On Tue, Jan 10, 2012 at 10:28 AM, Mark Petrovic wrote:
    Hello.

    This note is longer than I thought it would be, but I do not believe
    my situation is complicated.

    I am a RabbitMQ newcomer and am trying to identify which exchange type
    to use for my application.

    I have a publisher P that needs to send messages about database table
    updates. A message will bear the name of a single table name,
    followed by some opaque application data:

    Logically,

    message == tableName | somedata

    So consider a simplified database with three tables of interest: T1,
    T2, and T3.

    There are two consumers, C1 and C2.

    C1 needs to receive all messages concerning tables T1 and T2.

    C2 needs to receive all messages concerning tables T2 and T3.

    By implication, C1 and C2 must both receive messages about T2, where
    their interests overlap.

    Ideally, I want to publish to a single exchange, and, therefore, my
    consumers also bind to this same single exchange. I say this because
    in fact there are a lot more than three tables to treat - there are
    almost 200. A proliferation of exchanges per-table would be not good.

    I am considering a topic or header exchange.

    If a header exchange, I was thinking of the publisher P putting the
    concerned table name in a "table header" and the consumers binding to
    the exchange with an interest in receiving messages with a table
    header value equal to T1, T2, or T3 (I believe x-match == any would be
    appropriate when the consumer binds). But some of my readings on
    header exhchanges here
    http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2011-January/010935.html
    suggest that header exchanges may not be as useful as maybe the name
    suggests. Maybe I'm being overly paranoid.

    And I consider topic exchanges because I know they admit messages
    about T2 consumed by C1 still being available to C2.

    Would someone be kind enough to suggest approaches to discern which
    type of exchange to use.

    Thank you very kindly.

    --
    Mark


    --
    Mark


    --
    Mark
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120111/fb624872/attachment.htm>
  • Mark Petrovic at Jan 11, 2012 at 3:09 pm
    Thank you, Alvaro.

    Yes, I discovered that each consumer must have its own uniquely-named
    queue. I originally derived that queue name using a 128-bit UUID, but
    discovered later that if I pass an empty string for the queue name in
    the queue-declare, the server will generate a unique queue name for
    me. Perfect.

    I appreciate your kind response.

    I just downloaded the latest MEAP version of your book. It's a
    wonderful resource. Thank you!

    Mark
    On Wed, Jan 11, 2012 at 1:28 AM, Alvaro Videla wrote:
    Hi Mark,

    Yes a topic exchange can be ideal when you need to "listen" based on several
    patterns. Also this exchange leaves the possibility open for new table
    names,
    You just have to think about the binding pattern that you want to use.

    Also keep in mind that if you want message X to arrive at consumer A and B
    then you need one separate queue per consumer bound to that exchange using a
    binding key that matches the routing key used when publishing message X.
    What this means is that messages are "fanout'ed" at the exchange, but once
    they arrive at a queue then messages are pulled out individually and once
    gone from the queue is not seen again.

    Cheers,

    Alvaro
    On Wed, Jan 11, 2012 at 1:46 AM, Mark Petrovic wrote:

    I think I have a solution, better than one based on a header exchange
    or the scheme I outline below.

    It's basically this solution:

    http://www.rabbitmq.com/tutorials/tutorial-five-java.html

    where the routing keys are the table names. ?This gives me a single
    consumer queue bound to mulitple routing keys, which gets my job done.



    On Tue, Jan 10, 2012 at 2:20 PM, Mark Petrovic <mspetrovic at gmail.com>
    wrote:
    Talking to myself here (I have no problem with that :-)), I think I
    have at least one solution to this.

    - Create a single exchange E
    - The Publisher P creates a channel with a single queue and routing
    key with the same names as the Table T in question.
    - Publisher publishes a message to channel
    - The Consumer C1 iterates over the table names T1 and T2 he cares
    about and binds his channel to queues and routing keys with the same
    name as the table name iterated over
    - The Consumer C1 then starts a thread for each of these queues, and
    whose run() method calls channel.basicConsume(queueName, autoAck,
    rabbitConsumer)

    where rabbitConsumer extends com.rabbitmq.client.DefaultConsumer.

    While this seems to work, and is a bit easier to understand how to
    program than header exchanges (if a header exchange would even work
    for me - I wonder), I dont' like the thread-per-queue I need to spawn
    to handle messages inbound for each queue.

    What would be nice is if I could call channel.basicConsume(autoAck,
    rabbitConsumer), which would read messages off all the queues that
    were bound to that channel.

    While I have something working, I'm not yet real thrilled with my
    programming model.

    Anybody?

    Thanks!

    On Tue, Jan 10, 2012 at 10:28 AM, Mark Petrovic <mspetrovic at gmail.com>
    wrote:
    Hello.

    This note is longer than I thought it would be, but I do not believe
    my situation is complicated.

    I am a RabbitMQ newcomer and am trying to identify which exchange type
    to use for my application.

    I have a publisher P that needs to send messages about database table
    updates. ?A message will bear the name of a single table name,
    followed by some opaque application data:

    Logically,

    message == tableName | somedata

    So consider a simplified database with three tables of interest: ?T1,
    T2, and T3.

    There are two consumers, C1 and C2.

    C1 needs to receive all messages concerning tables T1 and T2.

    C2 needs to receive all messages concerning tables T2 and T3.

    By implication, C1 and C2 must both receive messages about T2, where
    their interests overlap.

    Ideally, I want to publish to a single exchange, and, therefore, my
    consumers also bind to this same single exchange. ?I say this because
    in fact there are a lot more than three tables to treat - there are
    almost 200. ?A proliferation of exchanges per-table would be not good.

    I am considering a topic or header exchange.

    If a header exchange, I was thinking of the publisher P putting the
    concerned table name in a "table header" and the consumers binding to
    the exchange with an interest in receiving messages with a table
    header value equal to T1, T2, or T3 (I believe x-match == any would be
    appropriate when the consumer binds). ?But some of my readings on
    header exhchanges here


    http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2011-January/010935.html

    suggest that header exchanges may not be as useful as maybe the name
    suggests. ?Maybe I'm being overly paranoid.

    And I consider topic exchanges because I know they admit messages
    about T2 consumed by C1 still being available to C2.

    Would someone be kind enough to suggest approaches to discern which
    type of exchange to use.

    Thank you very kindly.

    --
    Mark


    --
    Mark


    --
    Mark
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


    --
    Mark

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedJan 10, '12 at 6:28p
activeJan 11, '12 at 3:09p
posts5
users2
websiterabbitmq.com
irc#rabbitmq

2 users in discussion

Mark Petrovic: 4 posts Alvaro Videla: 1 post

People

Translate

site design / logo © 2022 Grokbase