FAQ
I have an instance which I tried upgrading from 3.0.2 to 3.1.0. Something broke during the upgrade and I booted a fresh new instance and connected my clients to it.


Now, I have valuable data in the 3.0.2 instance which I'd like to retrieve. If I could boot the instance, I'd shovel the data to the 3.1.0 instance.


I have attached the log files from the crashing instance.


Is there anything I can do to recuperate those messages?


Thanks,
Fran?ois Beausoleil


-------------- next part --------------
A non-text attachment was scrubbed...
Name: rabbit at rabbit104.log
Type: application/octet-stream
Size: 53604 bytes
Desc: not available
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130521/b9dfc756/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rabbit at rabbit104-sasl.log
Type: application/octet-stream
Size: 22497 bytes
Desc: not available
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130521/b9dfc756/attachment-0001.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: startup_err
Type: application/octet-stream
Size: 77 bytes
Desc: not available
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130521/b9dfc756/attachment-0002.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: startup_log
Type: application/octet-stream
Size: 22167 bytes
Desc: not available
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130521/b9dfc756/attachment-0003.obj>
-------------- next part --------------

Search Discussions

  • Simon MacMullen at May 22, 2013 at 9:44 am
    Hi Fran?ois.


    The stack trace you're seeing is consistent with ones we've seen before
    after something causes a bunch of extra zeroes to be written to the end
    of the queue index.


    Currently we're not absolutely sure what causes this but suspect the OS
    not completing a write during shutdown.


    The good news is that we have a fix for reading queue indices in this
    state. The bad news is that it can't go into the 3.1.x series since it
    requires a change to the on disc format for the queue index.


    However, this fix now is in the nightly releases (and will be in 3.2.0
    eventually). So I think you should be able to get a nightly build to
    boot with that data store. I'd take a backup first though :-) Once you
    do, you won't be able to go back to 3.1.x, but you should be able to get
    messages out to a 3.1.x broker with the shovel.


    Hope this helps.


    Cheers, Simon

    On 21/05/13 18:25, Fran?ois Beausoleil wrote:
    I have an instance which I tried upgrading from 3.0.2 to 3.1.0.
    Something broke during the upgrade and I booted a fresh new instance
    and connected my clients to it.

    Now, I have valuable data in the 3.0.2 instance which I'd like to
    retrieve. If I could boot the instance, I'd shovel the data to the
    3.1.0 instance.

    I have attached the log files from the crashing instance.

    Is there anything I can do to recuperate those messages?

    Thanks, Fran?ois Beausoleil







    _______________________________________________ rabbitmq-discuss
    mailing list rabbitmq-discuss at lists.rabbitmq.com
    https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss



    --
    Simon MacMullen
    RabbitMQ, Pivotal
  • François Beausoleil at May 22, 2013 at 2:57 pm
    Simon,


    Le 2013-05-22 ? 05:44, Simon MacMullen a ?crit :

    However, this fix now is in the nightly releases (and will be in 3.2.0 eventually). So I think you should be able to get a nightly build to boot with that data store. I'd take a backup first though :-) Once you do, you won't be able to go back to 3.1.x, but you should be able to get messages out to a 3.1.x broker with the shovel.

    You mean I should grab the latest from:


    http://www.rabbitmq.com/nightlies/rabbitmq-server/current/


    That still says "3.1.1".


    Thanks,
    Fran?ois
  • Simon MacMullen at May 22, 2013 at 3:05 pm

    On 22/05/13 15:57, Fran?ois Beausoleil wrote:
    You mean I should grab the latest from:

    http://www.rabbitmq.com/nightlies/rabbitmq-server/current/

    Yes.

    That still says "3.1.1".

    Well, we make the nightly version number out of (last released
    version).(serial number) so the current nightly build is 3.1.1.40522.
    But that doesn't make it part of the 3.1.x. series. Yeah, maybe we
    should change this. Hmm.


    Cheers, Simon


    --
    Simon MacMullen
    RabbitMQ, Pivotal
  • François Beausoleil at May 23, 2013 at 2:37 am

    Le 2013-05-22 ? 11:05, Simon MacMullen a ?crit :

    On 22/05/13 15:57, Fran?ois Beausoleil wrote:
    You mean I should grab the latest from:

    http://www.rabbitmq.com/nightlies/rabbitmq-server/current/

    Sorry to report RabbitMQ still doesn't start. I have the following crash using 3.1.1.40522:


    =CRASH REPORT==== 23-May-2013::02:33:49 ===
       crasher:
         initial call: application_master:init/4
         pid: <0.153.0>
         registered_name: []
         exception exit: {bad_return,
                          {{rabbit,start,[normal,[]]},
                           {'EXIT',
                            {rabbit,failure_during_boot,
                             {badmatch,
                              {error,
                               {{{function_clause,
                                  [{rabbit_queue_index,journal_minus_segment1,
                                    [{no_pub,no_del,ack},undefined]},
                                   {rabbit_queue_index,
                                    '-journal_minus_segment/2-fun-0-',4},
                                   {array,sparse_foldl_3,7},
                                   {array,sparse_foldl_2,9},
                                   {rabbit_queue_index,
                                    '-recover_journal/1-fun-0-',1},
                                   {dict,map_bucket,2},
                                   {dict,map_bucket,2},
                                   {dict,map_bkt_list,2}]},
                                 {gen_server2,call,[<0.223.0>,out,infinity]}},
                                {child,undefined,msg_store_persistent,
                                 {rabbit_msg_store,start_link,
                                  [msg_store_persistent,
                                   "/var/lib/rabbitmq/mnesia/rabbit at rabbit104",
                                   [<<251,36,244,99,192,144,85,200,209,96,141,61,
                                      212,212,28,95>>,
                                    <<191,32,57,94,174,211,166,0,76,193,74,149,252,
                                      177,185,245>>,
                                    <<212,228,229,81,23,60,150,249,91,90,197,56,156,
                                      20,130,194>>,
                                    <<120,60,75,235,213,77,54,137,14,124,111,62,250,
                                      138,144,114>>,
    ...


    startup_log says:


    Stack trace:
        [{rabbit_variable_queue,start_msg_store,2},
         {rabbit_amqqueue,recover,0},
         {rabbit,recover,0},
         {rabbit,'-run_boot_step/1-lc$^1/1-1-',1},
         {rabbit,run_boot_step,1},
         {rabbit,'-start/2-lc$^0/1-0-',1},
         {rabbit,start,2},
         {application_master,start_it_old,4}]


    Anything else I can do?


    Bye,
    Fran?ois
  • Simon MacMullen at May 23, 2013 at 10:59 am

    On 23/05/13 03:37, Fran?ois Beausoleil wrote:
    Sorry to report RabbitMQ still doesn't start. I have the following crash using 3.1.1.40522:

    Sorry, not sure what I was thinking. The new code will only recover if
    the corruption happens *after* the upgrade. Sorry to mislead.


    If the Mnesia dir is not too big and not too confidential, tar it up and
    send it to me off-list and I will see what I can do. (Preferably a copy
    which has not been through the upgrade process.)


    Cheers, Simon


    --
    Simon MacMullen
    RabbitMQ, Pivotal
  • François Beausoleil at May 23, 2013 at 11:38 am
    As you suggested, I backed up before. The tar is 2-3 GiB. My next step was exactly a fresh install then replace the mnesia dir with my backup. I'll let you know today.


    Fran?ois


    El 2013-05-23, a las 06:59, Simon MacMullen <simon@rabbitmq.com> escribi?:

    On 23/05/13 03:37, Fran?ois Beausoleil wrote:
    Sorry to report RabbitMQ still doesn't start. I have the following crash using 3.1.1.40522:
    Sorry, not sure what I was thinking. The new code will only recover if the corruption happens *after* the upgrade. Sorry to mislead.

    If the Mnesia dir is not too big and not too confidential, tar it up and send it to me off-list and I will see what I can do. (Preferably a copy which has not been through the upgrade process.)

    Cheers, Simon

    --
    Simon MacMullen
    RabbitMQ, Pivotal
  • Matthias Radestock at May 23, 2013 at 11:51 am
    Fran?ois,

    On 23/05/13 12:38, Fran?ois Beausoleil wrote:
    My next step was exactly a fresh install then replace the mnesia dir
    with my backup.

    That is going to produce the same result, so don't bother.


    Matthias.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedMay 21, '13 at 5:25p
activeMay 23, '13 at 11:51a
posts8
users3
websiterabbitmq.com
irc#rabbitmq

People

Translate

site design / logo © 2017 Grokbase