So I know from an instructions point of view how to set up a clustered
environment with disk nodes and ram nodes, but from the documentation/man
pages I am still unclear of a couple of things.
1. Assuming I have 3 nodes, node A, B, and C. Does node A become a disk
node by default when I set up the vhosts/users/permissions?

2. With the same node arrangement listed above.... I set up A(created vhost,
users, set permissions etc) and then clustered B against A,B (rabbitmqctl
cluster nodeA nodeB). I took down A and my messages did not persist even
though B should be a disk node. Is there something wrong with my steps?

3. What is the conventional/non-conventional way to determine whether or not
individual nodes in a cluster are a disk or ram nodes?

Thanks.

--
View this message in context: http://old.nabble.com/Disk-node-vs-ram-node-tp29356379p29356379.html
Sent from the RabbitMQ mailing list archive at Nabble.com.

Search Discussions

  • Alexandru Scvortov at Aug 5, 2010 at 7:43 pm

    1. Assuming I have 3 nodes, node A, B, and C. Does node A become a disk
    node by default when I set up the vhosts/users/permissions?
    The disk nodes are the ones explicitly listed in the "rabbitmqctl
    cluster" command.

    So,
    % rabbitmqctl cluster rabbit at A rabbit at B
    results in a cluster where A and B are disk nodes. If that command is
    run on either A or B, the cluster has just the two nodes.
    If, on the other hand, that command is run on C, you get a cluster with
    A and B disc nodes (same as above) and C as a ram node.
    2. With the same node arrangement listed above.... I set up A(created vhost,
    users, set permissions etc) and then clustered B against A,B (rabbitmqctl
    cluster nodeA nodeB). I took down A and my messages did not persist even
    though B should be a disk node. Is there something wrong with my steps?
    Queues exist physically only on the node on which they were declared.
    Clustering just makes them (and exchanges) accesible transparently on
    all of the nodes.

    Thus, if the queue was declared on A, and A was taken down, the messages
    stored on the queue would become inaccesible.

    If you want to never lose messages, have a look at our high-availability
    clustering guide in the documentation section of the website.
    3. What is the conventional/non-conventional way to determine whether or not
    individual nodes in a cluster are a disk or ram nodes?
    To find out whether nodes are disc or ram, run
    % rabbitmqctl status
    on any node in the cluster.

    Unfortunately, due to a bug, that command will not report the correct
    node type, so it's not of much help to you now. This problem has been
    fixed in our default branch and will be in the release.

    Cheers,
    Alex
  • Vince Chang at Aug 5, 2010 at 8:49 pm
    Thanks for your response Alex,

    1. So if I start node A by itself, and then cluster B against A, then A becomes a disk node because I specified A in the rabbitmqctl command? Here are the steps

    Node1:
    start_app
    add_vhost host1
    add_user user pass

    Node2:
    stop_app
    reset
    cluster node1 <--- (is this the step that makes node A a disk node?)
    start_app

    Node3:
    stop_app
    reset
    cluster node1
    start_app

    2. To be clear, if I send a message that creates the queue Q1 in node A in a cluster with node A, node B, and node C and node A goes down, then I will lose those messages on node A. Node A is down, but B and C should still be alive. If I send the same message to be queued on Q1, will Q1 be created on node B and node C? For me so far, that is not the case. When I send a message to be queued on Q1 (where the messages on Q1 on node A are now lost because node A is down) I do not see the queue getting created on either node B or node C. Is there something wrong with my steps?

    3. I am using RabbitMQ 1.7.2. Just to be clear there is _no way_ on 1.7.2 to determine whether or not a node is a disk node or ram node?

    -----Original Message-----
    From: Alexandru Scvor?ov [mailto:scvalex at gmail.com] On Behalf Of Alexandru Scvortov
    Sent: Thursday, August 05, 2010 12:43 PM
    To: Vince Chang
    Cc: rabbitmq-discuss at lists.rabbitmq.com
    Subject: Re: [rabbitmq-discuss] Disk node vs ram node
    1. Assuming I have 3 nodes, node A, B, and C. Does node A become a disk
    node by default when I set up the vhosts/users/permissions?
    The disk nodes are the ones explicitly listed in the "rabbitmqctl
    cluster" command.

    So,
    % rabbitmqctl cluster rabbit at A rabbit at B
    results in a cluster where A and B are disk nodes. If that command is
    run on either A or B, the cluster has just the two nodes.
    If, on the other hand, that command is run on C, you get a cluster with
    A and B disc nodes (same as above) and C as a ram node.
    2. With the same node arrangement listed above.... I set up A(created vhost,
    users, set permissions etc) and then clustered B against A,B (rabbitmqctl
    cluster nodeA nodeB). I took down A and my messages did not persist even
    though B should be a disk node. Is there something wrong with my steps?
    Queues exist physically only on the node on which they were declared.
    Clustering just makes them (and exchanges) accesible transparently on
    all of the nodes.

    Thus, if the queue was declared on A, and A was taken down, the messages
    stored on the queue would become inaccesible.

    If you want to never lose messages, have a look at our high-availability
    clustering guide in the documentation section of the website.
    3. What is the conventional/non-conventional way to determine whether or not
    individual nodes in a cluster are a disk or ram nodes?
    To find out whether nodes are disc or ram, run
    % rabbitmqctl status
    on any node in the cluster.

    Unfortunately, due to a bug, that command will not report the correct
    node type, so it's not of much help to you now. This problem has been
    fixed in our default branch and will be in the release.

    Cheers,
    Alex
  • Alexandru Scvortov at Aug 5, 2010 at 9:38 pm

    On Thu, Aug 05, 2010 at 01:49:51PM -0700, Vince Chang wrote:
    1. So if I start node A by itself, and then cluster B against A, then A becomes a disk node because I specified A in the rabbitmqctl command? Here are the steps

    Node1:
    start_app
    add_vhost host1
    add_user user pass

    Node2:
    stop_app
    reset
    cluster node1 <--- (is this the step that makes node A a disk node?)
    start_app

    Node3:
    stop_app
    reset
    cluster node1
    start_app
    I've just tried that on default.

    All the nodes start up as disc nodes. At that point, you turn *Node2*
    into a ram node. In the third group of commands, you also turn *Node3*
    into a ram node.

    So, the sequence is:
    1. Start Node1 as a disc node
    2. Cluster Node2 with Node1; Node2 becomes a ram node
    3. Cluster Node3 with Node1; Node3 becomes a ram node
    At the end, Node1 is disc; Node2 and Node3 are ram
    2. To be clear, if I send a message that creates the queue Q1 in node A in a cluster with node A, node B, and node C and node A goes down, then I will lose those messages on node A. Node A is down, but B and C should still be alive. If I send the same message to be queued on Q1, will Q1 be created on node B and node C? For me so far, that is not the case. When I send a message to be queued on Q1 (where the messages on Q1 on node A are now lost because node A is down) I do not see the queue getting created on either node B or node C. Is there something wrong with my steps?
    I'm not quite sure what you mean by ``message that creates the queue''.

    If you declare a queue Q1 on node A with queue.declare, then Q1 will
    reside on node A. If node A goes down, all the messages on its queues
    (including Q1) will become inaccessible. If Q1 was declared as durable,
    then the messages published as persistent on it will become available
    when node A comes back online.

    In a cluster, when a node goes down, its queues become inaccesbile.
    They are *not* automatically created on other nodes.

    Incidentally, which client are you using?
    3. I am using RabbitMQ 1.7.2. Just to be clear there is _no way_ on 1.7.2 to determine whether or not a node is a disk node or ram node?
    There's no way to do it with the provided tools.

    If you're not squeamish about running a bit of Erlang, you could do
    this:

    % erl -sname mgmtsh -remsh rabbit at localhost

    This gives you a remote Erlang shell named mgmtsh, connected to
    rabbit at localhost

    Then, running this should give you the disc nodes:
    (rabbit at localhost)1> mnesia:table_info(rabbit_durable_exchange, disc_copies).

    And running this should give you the ram nodes:
    (rabbit at localhost)2> mnesia:table_info(rabbit_durable_exchange, ram_copies).


    Cheers,
    Alex
  • Vince Chang at Aug 5, 2010 at 9:56 pm
    Thanks for your answers!

    #2. I mean that if I send a message that is meant for Q1, where Q1 was originally on node A but now is down because node A is down, will the queue be created on the other nodes B,C? It looks like your answer is no, just to confirm? I am using the 1.7.2 Java client. Is there a way to set up the cluster such that a queue (like Q1 in our case) will be created(automatically) on other nodes (such as node B and node C) if node A goes down? What happens when I make either (node B or node C) or both (node B and node C) disk nodes?

    #3. Not sure if I should do anything before hand, but I ran the command you provided and got this output:
    $ erl -sname mgmtsh -remsh rabbit at localhost
    Erlang R13B03 (erts-5.7.4) [source] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false]

    *** ERROR: Shell process terminated! (^G to start new job) ***
    {error_logger,{{2010,8,5},{14,52,59}},"~s~n",["Error in process <0.33.0> on node 'mgmtsh at ebony' with exit value: {badarg,[{erlang,list_to_existing_atom,[\"rabbit at ubuntu\"]},{dist_util,recv_challenge,1},{dist_util,handshake_we_started,1}]}\n"]}

    =ERROR REPORT==== 5-Aug-2010::14:52:59 ===
    Error in process <0.33.0> on node 'mgmtsh at ebony' with exit value: {badarg,[{erlang,list_to_existing_atom,["rabbit at ubuntu"]},{dist_util,recv_challenge,1},{dist_util,handshake_we_started,1}]}

    FYI: I'm running Ubuntu and installed rabbitMQ 1.7.2 with sudo apt-get install rabbitmq-server.

    -----Original Message-----
    From: Alexandru Scvor?ov [mailto:scvalex at gmail.com] On Behalf Of Alexandru Scvortov
    Sent: Thursday, August 05, 2010 2:38 PM
    To: Vince Chang
    Cc: rabbitmq-discuss at lists.rabbitmq.com
    Subject: Re: [rabbitmq-discuss] Disk node vs ram node
    On Thu, Aug 05, 2010 at 01:49:51PM -0700, Vince Chang wrote:
    1. So if I start node A by itself, and then cluster B against A, then A becomes a disk node because I specified A in the rabbitmqctl command? Here are the steps

    Node1:
    start_app
    add_vhost host1
    add_user user pass

    Node2:
    stop_app
    reset
    cluster node1 <--- (is this the step that makes node A a disk node?)
    start_app

    Node3:
    stop_app
    reset
    cluster node1
    start_app
    I've just tried that on default.

    All the nodes start up as disc nodes. At that point, you turn *Node2*
    into a ram node. In the third group of commands, you also turn *Node3*
    into a ram node.

    So, the sequence is:
    1. Start Node1 as a disc node
    2. Cluster Node2 with Node1; Node2 becomes a ram node
    3. Cluster Node3 with Node1; Node3 becomes a ram node
    At the end, Node1 is disc; Node2 and Node3 are ram
    2. To be clear, if I send a message that creates the queue Q1 in node A in a cluster with node A, node B, and node C and node A goes down, then I will lose those messages on node A. Node A is down, but B and C should still be alive. If I send the same message to be queued on Q1, will Q1 be created on node B and node C? For me so far, that is not the case. When I send a message to be queued on Q1 (where the messages on Q1 on node A are now lost because node A is down) I do not see the queue getting created on either node B or node C. Is there something wrong with my steps?
    I'm not quite sure what you mean by ``message that creates the queue''.

    If you declare a queue Q1 on node A with queue.declare, then Q1 will
    reside on node A. If node A goes down, all the messages on its queues
    (including Q1) will become inaccessible. If Q1 was declared as durable,
    then the messages published as persistent on it will become available
    when node A comes back online.

    In a cluster, when a node goes down, its queues become inaccesbile.
    They are *not* automatically created on other nodes.

    Incidentally, which client are you using?
    3. I am using RabbitMQ 1.7.2. Just to be clear there is _no way_ on 1.7.2 to determine whether or not a node is a disk node or ram node?
    There's no way to do it with the provided tools.

    If you're not squeamish about running a bit of Erlang, you could do
    this:

    % erl -sname mgmtsh -remsh rabbit at localhost

    This gives you a remote Erlang shell named mgmtsh, connected to
    rabbit at localhost

    Then, running this should give you the disc nodes:
    (rabbit at localhost)1> mnesia:table_info(rabbit_durable_exchange, disc_copies).

    And running this should give you the ram nodes:
    (rabbit at localhost)2> mnesia:table_info(rabbit_durable_exchange, ram_copies).


    Cheers,
    Alex
  • Dave Greggory at Aug 11, 2010 at 1:17 pm
    I'm a little concerned regarding what happens when a node that contains a certain queue in a cluster goes down.
    On Aug 5, 2010, at 5:38 PM, Alexandru Scvortov wrote:

    If you declare a queue Q1 on node A with queue.declare, then Q1 will
    reside on node A. If node A goes down, all the messages on its queues
    (including Q1) will become inaccessible. If Q1 was declared as durable,
    then the messages published as persistent on it will become available
    when node A comes back online.

    In a cluster, when a node goes down, its queues become inaccesbile.
    They are *not* automatically created on other nodes.
    I'm using the java client (1.8.1) and I automatically reconnect my producers/consumers to another node when a node goes down. I do this by setting up a shutdown listener hook on the connection.

    When a node with a queue goes down, can I write something using the java client that re-declares all known queues on the other node. May be have all the queue names(and params) registered somewhere in the app and upon reconnect, passively re-declare them. If passive declaration throws an exception, catch that and do an actual declaration. Would that work?

    If that works, then what happens if we bring up the node that went down that had the queue originally? Would it play nice?

    I'd like to find a solution without resorting to using the pacemaker approach.

    This is important not just for node failures but for other things as upgrading or reconfiguring rabbit without affecting any clients. (our rabbits sit behind a round-robin load balancer).
  • Dave Greggory at Aug 12, 2010 at 12:55 pm
    Maybe no one saw my questions below due to their placement in the original email. Anybody?
    When a node with a queue goes down, can I write something using the java client that re-declares all known queues on the other node. May be have all the queue names(and params) registered somewhere in the app and upon reconnect, passively re-declare them. If passive declaration throws an exception, catch that and do an actual declaration. Would that work?

    If that works, then what happens if we bring up the node that went down that had the queue originally? Would it play nice?
  • Alexandru Scvorţov at Aug 12, 2010 at 1:33 pm
    Hi Dave,
    When a node with a queue goes down, can I write something using the java client that re-declares all known queues on the other node. May be have all the queue names(and params) registered somewhere in the app and upon reconnect, passively re-declare them. If passive declaration throws an exception, catch that and do an actual declaration. Would that work?
    If you:
    1) declare a durable queue on one of the clustered nodes,
    2) take down that node,
    3) declare the same queue on another node
    you get a "404: Not Found".

    If it's not durable, sure.

    As an aside, if you passive declare something, then immediately declare
    it, the first step is redundant. There's no problem with declaring
    the same thing repeatedly. (assuming it's declared in exactly the
    same way and it's not unreachable)

    Cheers,
    Alex
  • Dave Greggory at Aug 13, 2010 at 2:36 pm
    Ouch... so no way to ensure that there's something to catch messages when the
    node with the original durable queue goes down?

    There are several mentions throughout the site about out-of-the-box live
    failover in future release, how far out is that? 6 months? 1 year?

    Not a big fan of the pacemaker approach due to the number of moving parts and
    different packages (pacemaker/corosync/heartbeat/erbd) to maintain and manage.
    And also haven't seen any indication of people actually using that setup in a
    complex VM environment in production.




    ----- Original Message ----
    From: Alexandru Scvor?ov <alexandru at rabbitmq.com>
    To: Dave Greggory <davegreggory at yahoo.com>
    Cc: "rabbitmq-discuss at lists.rabbitmq.com" <rabbitmq-discuss at lists.rabbitmq.com>
    Sent: Thu, August 12, 2010 9:33:54 AM
    Subject: Re: [rabbitmq-discuss] Disk node vs ram node

    Hi Dave,
    When a node with a queue goes down, can I write something using the java client

    that re-declares all known queues on the other node. May be have all the queue
    names(and params) registered somewhere in the app and upon reconnect, passively
    re-declare them. If passive declaration throws an exception, catch that and do
    an actual declaration. Would that work?
    If you:
    1) declare a durable queue on one of the clustered nodes,
    2) take down that node,
    3) declare the same queue on another node
    you get a "404: Not Found".

    If it's not durable, sure.

    As an aside, if you passive declare something, then immediately declare
    it, the first step is redundant. There's no problem with declaring
    the same thing repeatedly. (assuming it's declared in exactly the
    same way and it's not unreachable)

    Cheers,
    Alex
  • Alexis Richardson at Aug 13, 2010 at 2:40 pm
    Dave

    One of the things on our todo list, as you can imagine, is to
    understand how to best provide HA in a VM based setting. Some VM
    technologies provide useful failover / snapshot capability...

    alexis
    On Fri, Aug 13, 2010 at 3:36 PM, Dave Greggory wrote:


    Ouch... so no way to ensure that there's something to catch messages when the
    node with the original durable queue goes down?

    There are several mentions throughout the site about out-of-the-box live
    failover in future release, how far out is that? 6 months? 1 year?

    Not a big fan of the pacemaker approach due to the number of moving parts and
    different packages (pacemaker/corosync/heartbeat/erbd) to maintain and manage.
    And also haven't seen any indication of people actually using that setup in a
    complex VM environment in production.




    ----- Original Message ----
    From: Alexandru Scvor?ov <alexandru at rabbitmq.com>
    To: Dave Greggory <davegreggory at yahoo.com>
    Cc: "rabbitmq-discuss at lists.rabbitmq.com" <rabbitmq-discuss at lists.rabbitmq.com>
    Sent: Thu, August 12, 2010 9:33:54 AM
    Subject: Re: [rabbitmq-discuss] Disk node vs ram node

    Hi Dave,
    When a node with a queue goes down, can I write something using the java client

    that re-declares all known queues on the other node. May be have all the queue
    names(and params) registered somewhere in the app and upon reconnect, passively
    re-declare them. If passive declaration throws an exception, catch that and do
    an actual declaration. Would that work?
    If you:
    ?1) declare a durable queue on one of the clustered nodes,
    ?2) take down that node,
    ?3) declare the same queue on another node
    you get a "404: Not Found".

    If it's not durable, sure.

    As an aside, if you passive declare something, then immediately declare
    it, the first step is redundant. ?There's no problem with declaring
    the same thing repeatedly. (assuming it's declared in exactly the
    same way and it's not unreachable)

    Cheers,
    Alex



    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Dave Greggory at Aug 13, 2010 at 4:45 pm
    I understand, look forward to hearing about your plans for HA.
    On Aug 13, 2010, at 10:40 AM, Alexis Richardson wrote:

    Dave

    One of the things on our todo list, as you can imagine, is to
    understand how to best provide HA in a VM based setting. Some VM
    technologies provide useful failover / snapshot capability...

    alexis
    On Fri, Aug 13, 2010 at 3:36 PM, Dave Greggory wrote:


    Ouch... so no way to ensure that there's something to catch messages when the
    node with the original durable queue goes down?

    There are several mentions throughout the site about out-of-the-box live
    failover in future release, how far out is that? 6 months? 1 year?

    Not a big fan of the pacemaker approach due to the number of moving parts and
    different packages (pacemaker/corosync/heartbeat/erbd) to maintain and manage.
    And also haven't seen any indication of people actually using that setup in a
    complex VM environment in production.




    ----- Original Message ----
    From: Alexandru Scvor?ov <alexandru at rabbitmq.com>
    To: Dave Greggory <davegreggory at yahoo.com>
    Cc: "rabbitmq-discuss at lists.rabbitmq.com" <rabbitmq-discuss at lists.rabbitmq.com>
    Sent: Thu, August 12, 2010 9:33:54 AM
    Subject: Re: [rabbitmq-discuss] Disk node vs ram node

    Hi Dave,
    When a node with a queue goes down, can I write something using the java client

    that re-declares all known queues on the other node. May be have all the queue
    names(and params) registered somewhere in the app and upon reconnect, passively
    re-declare them. If passive declaration throws an exception, catch that and do
    an actual declaration. Would that work?
    If you:
    1) declare a durable queue on one of the clustered nodes,
    2) take down that node,
    3) declare the same queue on another node
    you get a "404: Not Found".

    If it's not durable, sure.

    As an aside, if you passive declare something, then immediately declare
    it, the first step is redundant. There's no problem with declaring
    the same thing repeatedly. (assuming it's declared in exactly the
    same way and it's not unreachable)

    Cheers,
    Alex



    _______________________________________________
    rabbitmq-discuss mailing list
    rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
  • Alexandru Scvortov at Aug 5, 2010 at 11:07 pm
    #2. I mean that if I send a message that is meant for Q1, where Q1 was originally on node A but now is down because node A is down, will the queue be created on the other nodes B,C? It looks like your answer is no, just to confirm? I am using the 1.7.2 Java client. Is there a way to set up the cluster such that a queue (like Q1 in our case) will be created(automatically) on other nodes (such as node B and node C) if node A goes down? What happens when I make either (node B or node C) or both (node B and node C) disk nodes?
    There's no automatic way to create missing queues inside a cluster in
    RabbitMQ, as far as I know.

    Whether a node is disc or ram refers to where the broker metadata
    (queues, exchanges, bindings) is kept. Persistent messages are always
    stored on disk. When a ram node goes down, it looses all its metadata.
    If it's in a cluster, it will just connect to some other node and
    recover the information. If the entire cluster goes down (say, power
    outage), the metadata can only be recovered if at least one node was
    disc. In general, there's not much sense in having a lot of disc nodes.
    One or two should be enough. Ram nodes are faster.
    #3. Not sure if I should do anything before hand, but I ran the command you provided and got this output:
    $ erl -sname mgmtsh -remsh rabbit at localhost
    Erlang R13B03 (erts-5.7.4) [source] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false]

    *** ERROR: Shell process terminated! (^G to start new job) ***
    {error_logger,{{2010,8,5},{14,52,59}},"~s~n",["Error in process <0.33.0> on node 'mgmtsh at ebony' with exit value: {badarg,[{erlang,list_to_existing_atom,[\"rabbit at ubuntu\"]},{dist_util,recv_challenge,1},{dist_util,handshake_we_started,1}]}\n"]}

    =ERROR REPORT==== 5-Aug-2010::14:52:59 ===
    Error in process <0.33.0> on node 'mgmtsh at ebony' with exit value: {badarg,[{erlang,list_to_existing_atom,["rabbit at ubuntu"]},{dist_util,recv_challenge,1},{dist_util,handshake_we_started,1}]}

    FYI: I'm running Ubuntu and installed rabbitMQ 1.7.2 with sudo apt-get install rabbitmq-server.
    I assumed the rabbit at localhost was your node. Try rabbit at ebony.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedAug 5, '10 at 6:08p
activeAug 13, '10 at 4:45p
posts12
users4
websiterabbitmq.com
irc#rabbitmq

People

Translate

site design / logo © 2017 Grokbase