Hey all,

Wanting to add federation to existing brokers but not sure I'm
understanding the semantics involved.

Let's say I have three exchanges, {events, topic}, {control, direct},
and {config, topic}, all three of which are on both brokers (in
different datacenters).

My current rabbit.config looks like:

[{rabbitmq_federation
,[{exchanges, [
[{exchange, "config"}
,{type, "topic"}
,{upstream_set, "config_exchanges"}
]
,[{exchange, "control"}
,{type, "direct"}
,{upstream_set, "control_exchanges"}
]
,[{exchange, "events"}
,{type, "topic"}
,{upstream_set, "events_exchanges"}
]
]
,{upstream_sets, [{"control_exchanges", [[{connection,
"broker2"}]]}
,{"events_exchanges", [[{connection, "broker2"}]]}
,{"config_exchanges", [[{connection, "broker2"}]]}
]}
,{connections, [{"broker2", [{host, "broker2.example.com"}]}]}
,{local_nodename, "broker1.example.com"}
]}
...more config


I have an almost identical config on brokers 1 and 2 (with hostnames
obviously adapted). Both startup fine (*sidenote: I had to set
ERL_LIBS=/path/to/rabbit/plugins to get the federation plugin to work -
the VM couldn't find the amqp_connection module, even after enabling the
amqp_client plugin). I see both brokers communicating about the
exchanges - lines similar to:

=INFO REPORT==== 26-Nov-2011::18:04:01 ===
Federation exchange 'control' in vhost '/' connected to
broker2.example.com:/:control

I start my application (in a separate VM) connecting to broker1, try to
declare my exchanges as part of my app's init, and get errors like:

...snip...
{server_initiated_close,406,
<<"PRECONDITION_FAILED - cannot redeclare exchange
'control' in vhost '/' with different type, durable, internal or
autodelete value">>}},
{gen_server,call,
[<0.78.0>,
{call,
{'exchange.declare',1,<<"control">>,<<"direct">>,
false,false,false,false,false,[]},
...snip...

So I explicitly set type, durable, internal, and autodelete to 'false'
in my federation configs, but to no avail. I should note I have no
durable exchanges or queues in this system; everything is ephemeral.
Neither broker is clustered with any others.

I do see errors when I restart a broker similar to:

=ERROR REPORT==== 26-Nov-2011::18:04:00 ===
connection <0.2064.0>, channel 2 - error:
{amqp_error,not_found,
"no exchange 'federation: control -> broker2.example.com B'
in vhost '/'",
'exchange.delete'}

My understanding (which may be bass-ackwards) is that by setting up the
'control' exchange to federate with broker2, broker1 will receive
payloads from broker2's 'control' exchange and route them to any queues
bound to broker1's 'control' exchange, and that my application code
remains relatively unchanged to take advantage of that. The converse
too: since broker2 is connecting to broker1, any payloads my apps on
broker1 send to the 'control' exchange will be relayed to broker2.

Thanks for any guidance; sorry for the wall of text.

James

PS: My broker output when starting up:

Activating RabbitMQ plugins ...
2 plugins activated:
* amqp_client-2.7.0
* rabbitmq_federation-2.7.0


+---+ +---+
+---+ +-------+
RabbitMQ +---+ |
v2.7.0 +---+ |
+-------------------+
AMQP 0-9-1 / 0-9 / 0-8
Copyright (C) 2007-2011 VMware, Inc.
Licensed under the MPL. See http://www.rabbitmq.com/

node : rabbit at broker1.example.com
app descriptor :
/var/lib/rabbitmq/rabbitmq_server-2.7.0/sbin/../ebin/rabbit.app
home dir : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config
cookie hash : SgcmwZTTAvznNXVC3eUfkQ==
log : /var/log/rabbitmq/rabbit at broker1.example.com.log
sasl log : /var/log/rabbitmq/rabbit at broker1.example.com-sasl.log
database dir : /var/lib/rabbitmq/mnesia/rabbit at broker1.example.com
erlang version : 5.8.4

- -- rabbit boot start
starting file handle cache server
...done
starting worker pool
...done
starting database
...done
starting codec correctness check
...done
- -- external infrastructure ready
starting plugin registry
...done
starting auth mechanism cr-demo
...done
starting auth mechanism amqplain
...done
starting auth mechanism plain
...done
starting statistics event manager
...done
starting logging server
...done
starting exchange type direct
...done
starting exchange type fanout
...done
starting exchange type headers
...done
starting exchange type topic
...done
- -- kernel ready
starting alarm handler
...done
starting node monitor
...done
starting cluster delegate
...done
starting guid generator
...done
starting memory monitor
...done
- -- core initialized
starting empty DB check
...done
starting federation
...done
starting federation exchange type
...done
starting exchange, queue and binding recovery
...done
starting mirror queue slave sup
...done
starting adding mirrors to queues
...done
- -- message delivery logic ready
starting error log relay
...done
starting networking
...done
starting direct_client
...done
starting notify cluster nodes
...done

broker running


- --
James Aimonetti
Distributed Systems Engineer / DJ MC_

2600hz | http://2600hz.com
sip:james at 2600hz.com
tel: 415.886.7905

Search Discussions

  • Simon MacMullen at Nov 28, 2011 at 10:56 am

    On 26/11/11 18:27, James Aimonetti wrote:
    Wanting to add federation to existing brokers but not sure I'm
    understanding the semantics involved.
    Hi. Thanks for the reminder that an upgrade to the federation docs is
    overdue :)
    Let's say I have three exchanges, {events, topic}, {control, direct},
    and {config, topic}, all three of which are on both brokers (in
    different datacenters).

    My current rabbit.config looks like:

    [{rabbitmq_federation
    ,[{exchanges, [
    [{exchange, "config"}
    ,{type, "topic"}
    ,{upstream_set, "config_exchanges"}
    ]
    ,[{exchange, "control"}
    ,{type, "direct"}
    ,{upstream_set, "control_exchanges"}
    ]
    ,[{exchange, "events"}
    ,{type, "topic"}
    ,{upstream_set, "events_exchanges"}
    ]
    ]
    ,{upstream_sets, [{"control_exchanges", [[{connection,
    "broker2"}]]}
    ,{"events_exchanges", [[{connection, "broker2"}]]}
    ,{"config_exchanges", [[{connection, "broker2"}]]}
    ]}
    ,{connections, [{"broker2", [{host, "broker2.example.com"}]}]}
    ,{local_nodename, "broker1.example.com"}
    ]}
    ...more config
    For the record you don't need to have a different upstream_set for each
    exchange, unless you actually want each exchange to federate with
    different brokers.
    I have an almost identical config on brokers 1 and 2 (with hostnames
    obviously adapted). Both startup fine (*sidenote: I had to set
    ERL_LIBS=/path/to/rabbit/plugins to get the federation plugin to work -
    the VM couldn't find the amqp_connection module, even after enabling the
    amqp_client plugin).
    Really? Are you using deb / rpms? Have you moved anything around?
    I see both brokers communicating about the
    exchanges - lines similar to:

    =INFO REPORT==== 26-Nov-2011::18:04:01 ===
    Federation exchange 'control' in vhost '/' connected to
    broker2.example.com:/:control

    I start my application (in a separate VM) connecting to broker1, try to
    declare my exchanges as part of my app's init, and get errors like:
    You don't need to. The exchanges part of the federation config will
    declare those exchanges when the broker starts up.

    If you do want to declare them in your app, read the section "Declaring
    Federation Exchanges Over AMQP" in the README.
    ...snip...
    {server_initiated_close,406,
    <<"PRECONDITION_FAILED - cannot redeclare exchange
    'control' in vhost '/' with different type, durable, internal or
    autodelete value">>}},
    {gen_server,call,
    [<0.78.0>,
    {call,
    {'exchange.declare',1,<<"control">>,<<"direct">>,
    false,false,false,false,false,[]},
    ...snip...

    So I explicitly set type, durable, internal, and autodelete to 'false'
    in my federation configs, but to no avail.
    The problem is likely to be the type - they need to be of type
    "x-federation". You might want to temporarily enable the management
    plugin so you can see what's going on.
    I should note I have no
    durable exchanges or queues in this system; everything is ephemeral.
    Neither broker is clustered with any others.

    I do see errors when I restart a broker similar to:

    =ERROR REPORT==== 26-Nov-2011::18:04:00 ===
    connection<0.2064.0>, channel 2 - error:
    {amqp_error,not_found,
    "no exchange 'federation: control -> broker2.example.com B'
    in vhost '/'",
    'exchange.delete'}
    Yeah, these are harmless - federation uses an intermediate exchange on
    the upstream which gets recreated on connect. It has to try to delete it
    to make *sure* it's not there (although it probably isn't anyway) - but
    there's currently no way to not treat that as an error.
    My understanding (which may be bass-ackwards) is that by setting up the
    'control' exchange to federate with broker2, broker1 will receive
    payloads from broker2's 'control' exchange and route them to any queues
    bound to broker1's 'control' exchange, and that my application code
    remains relatively unchanged to take advantage of that. The converse
    too: since broker2 is connecting to broker1, any payloads my apps on
    broker1 send to the 'control' exchange will be relayed to broker2.
    Yes, although the links are unidirectional so you need to set up
    federation in both directions.

    Cheers, Simon

    --
    Simon MacMullen
    RabbitMQ, VMware
  • James Aimonetti at Nov 28, 2011 at 7:17 pm
    Simon,

    Thanks for the info.

    My app will be running in environments that aren't necessarily federated, which is why I have my app declaring the exchanges on startup. Do you have recommendations for how to handle this? If federation is not enabled, and i declare the exchange as 'x-federation' with type='direct' in the args, will RabbitMQ know to declare the exchange as direct and not federated?

    As for the amqp_client/ERL_LIBS issue, I installed via tarball. Not sure why starting up didn't have plugins/ in the code path. I'll check the installed Erlang lib and see if a conflicting version made it in there.

    James

    James Aimonetti
    http://2600hz.com
    email: james at 2600hz.com
    work: 415.886.7905

    ________________________________________
    From: rabbitmq-discuss-bounces at lists.rabbitmq.com [rabbitmq-discuss-bounces at lists.rabbitmq.com] on behalf of Simon MacMullen [simon at rabbitmq.com]
    Sent: Monday, November 28, 2011 02:56
    To: rabbitmq-discuss at lists.rabbitmq.com
    Subject: Re: [rabbitmq-discuss] Federation help
    On 26/11/11 18:27, James Aimonetti wrote:
    Wanting to add federation to existing brokers but not sure I'm
    understanding the semantics involved.
    Hi. Thanks for the reminder that an upgrade to the federation docs is
    overdue :)
    Let's say I have three exchanges, {events, topic}, {control, direct},
    and {config, topic}, all three of which are on both brokers (in
    different datacenters).

    My current rabbit.config looks like:

    [{rabbitmq_federation
    ,[{exchanges, [
    [{exchange, "config"}
    ,{type, "topic"}
    ,{upstream_set, "config_exchanges"}
    ]
    ,[{exchange, "control"}
    ,{type, "direct"}
    ,{upstream_set, "control_exchanges"}
    ]
    ,[{exchange, "events"}
    ,{type, "topic"}
    ,{upstream_set, "events_exchanges"}
    ]
    ]
    ,{upstream_sets, [{"control_exchanges", [[{connection,
    "broker2"}]]}
    ,{"events_exchanges", [[{connection, "broker2"}]]}
    ,{"config_exchanges", [[{connection, "broker2"}]]}
    ]}
    ,{connections, [{"broker2", [{host, "broker2.example.com"}]}]}
    ,{local_nodename, "broker1.example.com"}
    ]}
    ...more config
    For the record you don't need to have a different upstream_set for each
    exchange, unless you actually want each exchange to federate with
    different brokers.
    I have an almost identical config on brokers 1 and 2 (with hostnames
    obviously adapted). Both startup fine (*sidenote: I had to set
    ERL_LIBS=/path/to/rabbit/plugins to get the federation plugin to work -
    the VM couldn't find the amqp_connection module, even after enabling the
    amqp_client plugin).
    Really? Are you using deb / rpms? Have you moved anything around?
    I see both brokers communicating about the
    exchanges - lines similar to:

    =INFO REPORT==== 26-Nov-2011::18:04:01 ===
    Federation exchange 'control' in vhost '/' connected to
    broker2.example.com:/:control

    I start my application (in a separate VM) connecting to broker1, try to
    declare my exchanges as part of my app's init, and get errors like:
    You don't need to. The exchanges part of the federation config will
    declare those exchanges when the broker starts up.

    If you do want to declare them in your app, read the section "Declaring
    Federation Exchanges Over AMQP" in the README.
    ...snip...
    {server_initiated_close,406,
    <<"PRECONDITION_FAILED - cannot redeclare exchange
    'control' in vhost '/' with different type, durable, internal or
    autodelete value">>}},
    {gen_server,call,
    [<0.78.0>,
    {call,
    {'exchange.declare',1,<<"control">>,<<"direct">>,
    false,false,false,false,false,[]},
    ...snip...

    So I explicitly set type, durable, internal, and autodelete to 'false'
    in my federation configs, but to no avail.
    The problem is likely to be the type - they need to be of type
    "x-federation". You might want to temporarily enable the management
    plugin so you can see what's going on.
    I should note I have no
    durable exchanges or queues in this system; everything is ephemeral.
    Neither broker is clustered with any others.

    I do see errors when I restart a broker similar to:

    =ERROR REPORT==== 26-Nov-2011::18:04:00 ===
    connection<0.2064.0>, channel 2 - error:
    {amqp_error,not_found,
    "no exchange 'federation: control -> broker2.example.com B'
    in vhost '/'",
    'exchange.delete'}
    Yeah, these are harmless - federation uses an intermediate exchange on
    the upstream which gets recreated on connect. It has to try to delete it
    to make *sure* it's not there (although it probably isn't anyway) - but
    there's currently no way to not treat that as an error.
    My understanding (which may be bass-ackwards) is that by setting up the
    'control' exchange to federate with broker2, broker1 will receive
    payloads from broker2's 'control' exchange and route them to any queues
    bound to broker1's 'control' exchange, and that my application code
    remains relatively unchanged to take advantage of that. The converse
    too: since broker2 is connecting to broker1, any payloads my apps on
    broker1 send to the 'control' exchange will be relayed to broker2.
    Yes, although the links are unidirectional so you need to set up
    federation in both directions.

    Cheers, Simon

    --
    Simon MacMullen
    RabbitMQ, VMware
    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Simon MacMullen at Nov 29, 2011 at 11:01 am

    On 28/11/11 19:17, James Aimonetti wrote:
    My app will be running in environments that aren't necessarily
    federated, which is why I have my app declaring the exchanges on
    startup. Do you have recommendations for how to handle this? If
    federation is not enabled, and i declare the exchange as
    'x-federation' with type='direct' in the args, will RabbitMQ know to
    declare the exchange as direct and not federated?
    Ah, no.

    Probably the best thing to do is require the federation plugin to always
    be installed, and in the non-federated environments declare your
    upstream_set(s) empty. This does require a bit of configuration on the
    non-federated environments though. I'll have a think about whether this
    can be improved.
    As for the amqp_client/ERL_LIBS issue, I installed via tarball. Not
    sure why starting up didn't have plugins/ in the code path. I'll
    check the installed Erlang lib and see if a conflicting version made
    it in there.
    Thanks.

    Cheers, Simon

    --
    Simon MacMullen
    RabbitMQ, VMware
  • James Aimonetti at Dec 11, 2011 at 5:11 pm
    Hi Simon,

    Not to be a bother or anything, but have you thought more about how to
    handle the case where an application won't necessarily know if the
    broker is using federation or not, and how the application can properly
    declare the exchange?

    We unfortunately can't ensure the exchanges are properly declared on
    startup, so the application makes a point to declare them on init. If I
    can declare it as federated and have the broker kick back an easily
    parse-able error indicating to not use the federation exchange type, and
    redeclare using the vanilla declaration, that would get me most of the
    way there.

    Happy holidays,

    James

    On 11/29/2011 03:01 AM, Simon MacMullen wrote:
    On 28/11/11 19:17, James Aimonetti wrote:
    My app will be running in environments that aren't necessarily
    federated, which is why I have my app declaring the exchanges on
    startup. Do you have recommendations for how to handle this? If
    federation is not enabled, and i declare the exchange as
    'x-federation' with type='direct' in the args, will RabbitMQ know to
    declare the exchange as direct and not federated?
    Ah, no.

    Probably the best thing to do is require the federation plugin to always
    be installed, and in the non-federated environments declare your
    upstream_set(s) empty. This does require a bit of configuration on the
    non-federated environments though. I'll have a think about whether this
    can be improved.
    As for the amqp_client/ERL_LIBS issue, I installed via tarball. Not
    sure why starting up didn't have plugins/ in the code path. I'll
    check the installed Erlang lib and see if a conflicting version made
    it in there.
    Thanks.

    Cheers, Simon

    - --
    James Aimonetti
    Distributed Systems Engineer / DJ MC_

    2600hz | http://2600hz.com
    sip:james at 2600hz.com
    tel: 415.886.7905
  • Simon MacMullen at Dec 12, 2011 at 12:24 pm
    You could either

    1) Not enable federation on non federated servers. Declare the exchange
    as type x-federation. If the connection closes with 503 COMMAND_INVALID
    back, then reestablish the connection and declare as direct (or whatever).

    2) Enable the federation plugin on all nodes, and configure the ones you
    don't want to *really* federate with empty upstream_sets.

    I would go for 2) personally.

    Cheers, Simon
    On 11/12/11 17:11, James Aimonetti wrote:
    -----BEGIN PGP SIGNED MESSAGE-----
    Hash: SHA1

    Hi Simon,

    Not to be a bother or anything, but have you thought more about how to
    handle the case where an application won't necessarily know if the
    broker is using federation or not, and how the application can properly
    declare the exchange?

    We unfortunately can't ensure the exchanges are properly declared on
    startup, so the application makes a point to declare them on init. If I
    can declare it as federated and have the broker kick back an easily
    parse-able error indicating to not use the federation exchange type, and
    redeclare using the vanilla declaration, that would get me most of the
    way there.

    Happy holidays,

    James

    On 11/29/2011 03:01 AM, Simon MacMullen wrote:
    On 28/11/11 19:17, James Aimonetti wrote:
    My app will be running in environments that aren't necessarily
    federated, which is why I have my app declaring the exchanges on
    startup. Do you have recommendations for how to handle this? If
    federation is not enabled, and i declare the exchange as
    'x-federation' with type='direct' in the args, will RabbitMQ know to
    declare the exchange as direct and not federated?
    Ah, no.

    Probably the best thing to do is require the federation plugin to always
    be installed, and in the non-federated environments declare your
    upstream_set(s) empty. This does require a bit of configuration on the
    non-federated environments though. I'll have a think about whether this
    can be improved.
    As for the amqp_client/ERL_LIBS issue, I installed via tarball. Not
    sure why starting up didn't have plugins/ in the code path. I'll
    check the installed Erlang lib and see if a conflicting version made
    it in there.
    Thanks.

    Cheers, Simon

    - --
    James Aimonetti
    Distributed Systems Engineer / DJ MC_

    2600hz | http://2600hz.com
    sip:james at 2600hz.com
    tel: 415.886.7905
    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.10 (GNU/Linux)
    Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

    iQEcBAEBAgAGBQJO5ORaAAoJENc77s1OYoGgVdIH/1IEHp7dZNQ73BF31tL86Iw3
    b507jpGkBajr4l2tizTVyF1j93FRl75bI2WDgSyb52DRXJs4eQcuJy7Csu2olM9l
    873T60tHSa0t0ID7PI4xJ55X6YdkL63sf4h6zUgq6jQMCbNl2GA2x6aBLNNA06YE
    WqiKoOM2Lyh2p9zPZs63h9zbl4aLdeITESvCzDTh7YiVXXYMeV6mgeYX5by+LonQ
    vhc8O8xoGH3cMYzKWDMcuxTFsEe6urMFevEIxIKNWsj7G9s/pH7YgfjoRiQmmkZS
    FyFrsK+mcYDsHRB5P/nJtiMc9C34Qp+wdW/y/vAlrtEAmZ4Bj2qYwCCw0Ov7dj8=
    =GyBb
    -----END PGP SIGNATURE-----

    --
    Simon MacMullen
    RabbitMQ, VMware

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedNov 26, '11 at 6:27p
activeDec 12, '11 at 12:24p
posts6
users2
websiterabbitmq.com
irc#rabbitmq

People

Translate

site design / logo © 2022 Grokbase