FAQ
Brett Cameron <brett.r.cameron@...> writes:



Karl,Wild speculation would be that messages have been consumed from a
queue but that queue (name = document_queue?) is for some reason missing in
action (gone, inaccessible, ...) when the consumer subsequently tries to ack
the message(s).
Brett


On Sat, May 18, 2013 at 3:20 PM, Karl Rieb <karl.rieb <at> gmail.com> wrote:Hi,
I am running a RabbitMQ cluster of 4 nodes in AWS EC2 on c1.xlarge
instances using Ubuntu 12.04 LTS (kernel 3.2.0-38). ?Periodically two of my
nodes holding the largest queues will crash and require me to restart the
rabbitmq-server service. ?Looking at the logs, I'm at a loss as to why the
crash is occurring. ?I've have tried reproducing the crash manually by
flooding my nodes with messages, but that doesn't seem to trigger the issue.
?Below is output from various RabbitMQ logs:
# cat /var/log/rabbitmq/rabbit <at> ip-10-123-123-123-sasl.log
=CRASH REPORT==== 18-May-2013::01:41:51 ===? crasher:? ? initial call:
gen:init_it/6? ? pid: <0.301.0>? ? registered_name: []? ? exception exit:
{{case_clause,{empty,{[],[]}}},? ? ? ? ? ? ? ? ? ?
?[{rabbit_amqqueue_process,subtract_acks,3,[]},
? ? ? ? ? ? ? ? ? ? ? {rabbit_amqqueue_process,subtract_acks,4,[]},? ? ? ?
? ? ? ? ? ? ? {rabbit_amqqueue_process,handle_cast,2,[]},? ? ? ? ? ? ? ? ? ?
? {gen_server2,handle_msg,2,[]},? ? ? ? ? ? ? ? ? ? ? {proc_lib,wake_up,3,
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [{file,"proc_lib.erl"},{line,249}]}]}? ? ?
in function ?gen_server2:terminate/3? ? ancestors:
[rabbit_amqqueue_sup,rabbit_sup,<0.207.0>]? ? messages: [{'$gen_cast',
{resume,<0.676.0>}},
? ? ? ? ? ? ? ? ? {'$gen_cast',{ack,[27583,27606,27629,27652,27675,27698,?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?27721,27744,27767,27790,27813,27836,? ? ?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?27859,27882,27905,27928,27951,27974,
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?27997,28020,28043,28066,28089,28112,?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?28135,28158,28179,28200,28221,28242,? ? ?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?28263,28284,28305,28324,28342,28360,
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?28378,28396,28414,28432,28450,28468,?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?28486,28504,28522,28540,28558,28576,? ? ?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?28594,28612],? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
? ? <0.737.0>}},
? ? ? ? ? ? ? ? ? {'$gen_cast',{resume,<0.737.0>}},? ? ? ? ? ? ? ? ?
{'$gen_cast',{ack,[27608,27631,27654,27677,27700,27723,? ? ? ? ? ? ? ? ? ? ?
? ? ? ? ? ? ? ?27746,27769,27792,27815,27838,27861,
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?27884,27907,27930,27953,27976,27999,?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?28022,28045,28068,28091,28114,28137,? ? ?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?28160,28181,28202,28223,28244,28265,
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?28286,28307,28326,28344,28362,28380,?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?28398,28416,28434,28452,28470,28488,? ? ?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?28506,28524,28542,28560,28578,28596,
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?28614],? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
? ? ? <0.676.0>}},? ? ? ? ? ? ? ? ? {'$gen_cast',{ack,
[27578,27601,27624,27647,27670,27693,? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
?27716,27739,27762,27785,27808,27831,
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?27854,27877,27900,27923,27946,27969,?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?27992,28015,28038,28061,28084,28107,? ? ?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?28130,28153,28174,28195,28216,28237,
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?28258,28279,28300,28321,28340,28358,?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?28376,28394,28412,28430,28448,28466,? ? ?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?28484,28502,28520,28538,28556,28574,
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?28592,28610],? ? ? ? ? ? ? ? ? ? ? ?
? ? ? ? ? ? <0.716.0>}},? ? ? ? ? ? ? ? ? {'$gen_cast',{resume,<0.716.0>}},?
? ? ? ? ? ? ? ? {'$gen_cast',{ack,[27582,27605,27628,27651,27674,27697,
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?27720,27743,27766,27789,27812,27835,?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?27858,27881,27904,27927,27950,27973,? ? ?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?27996,28019,28042,28065,28088,28111,
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?28134,28157,28178,28199,28220,28241,?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?28262,28283,28304,28323,28341,28359,? ? ?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?28377,28395,28413,28431,28449,28467,
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?28485,28503,28521,28539,28557,28575,?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?28593,28611],? ? ? ? ? ? ? ? ? ? ? ? ? ?
? ? ? ? <0.737.0>}}]? ? links: [<0.300.0>]
? ? dictionary: [{{ch,<0.729.0>},? ? ? ? ? ? ? ? ? ?{cr,
<0.729.0>,#Ref<0.0.0.12296>,? ? ? ? ? ? ? ? ? ? ?
?{[28715,28696,28677,28658,28639,28620,28602,28584,? ? ? ? ? ? ? ? ? ? ? ?
?28566,28548,28530,28512,28494,28476,28458,28440,
? ? ? ? ? ? ? ? ? ? ? ? ?28422,28404,28386,28368,28350,28332,28313,28292,?
? ? ? ? ? ? ? ? ? ? ? ?28271,28250,28229,28208,28187,28166,28143,28120,? ? ?
? ? ? ? ? ? ? ? ? ?28097,28074,28051,28028,28005,27982,27959,27936,
? ? ? ? ? ? ? ? ? ? ? ? ?27913,27890,27867,27844,27821,27798,27775,27752,?
? ? ? ? ? ? ? ? ? ? ? ?27729,27706,27683,27660,27637,27614],? ? ? ? ? ? ? ?
? ? ? ? [27591]},? ? ? ? ? ? ? ? ? ? ? ?1,? ? ? ? ? ? ? ? ? ? ? ?{[],[]},
? ? ? ? ? ? ? ? ? ? ? ?{qstate,<0.728.0>,active,{0,nil}},? ? ? ? ? ? ? ? ?
? ? ?11}},? ? ? ? ? ? ? ? ? {{credit_to,<7350.877.0>},22},? ? ? ? ? ? ? ? ?
{{ch,<0.608.0>},? ? ? ? ? ? ? ? ? ?{cr,<0.608.0>,#Ref<0.0.0.9041>,
? ? ? ? ? ? ? ? ? ? ? ?{[28708,28689,28670,28651,28632,28607,28589,28571,?
? ? ? ? ? ? ? ? ? ? ? ?28553,28535,28517,28499,28481,28463,28445,28427,? ? ?
? ? ? ? ? ? ? ? ? ?28409,28391,28373,28355,28337,28318,28297,28276,
? ? ? ? ? ? ? ? ? ? ? ? ?28255,28234,28213,28192,28171,28150,28127,28104,?
? ? ? ? ? ? ? ? ? ? ? ?28081,28058,28035,28012,27989,27966,27943,27920,? ? ?
? ? ? ? ? ? ? ? ? ?27897,27874,27851,27828,27805,27782,27759,27736,
? ? ? ? ? ? ? ? ? ? ? ? ?27713,27690,27667,27644,27621,27598],? ? ? ? ? ?
? ? ? ? ? ? [27575]},? ? ? ? ? ? ? ? ? ? ? ?1,? ? ? ? ? ? ? ? ? ? ? ?{[],
[]},? ? ? ? ? ? ? ? ? ? ? ?{qstate,<0.607.0>,active,{0,nil}},
? ? ? ? ? ? ? ? ? ? ? ?38}},? ? ? ? ? ? ? ? ? {{ch,<0.624.0>},? ? ? ? ? ?
? ? ? ?{cr,<0.624.0>,#Ref<0.0.0.9436>,? ? ? ? ? ? ? ? ? ? ?
?{[28722,28703,28684,28665,28646],[28627]},? ? ? ? ? ? ? ? ? ? ? ?1,
? ? ? ? ? ? ? ? ? ? ? ?{[],[]},? ? ? ? ? ? ? ? ? ? ? ?{qstate,
<0.623.0>,active,{0,nil}},? ? ? ? ? ? ? ? ? ? ? ?36}},? ? ? ? ? ? ? ? ?
{{ch,<0.696.0>},? ? ? ? ? ? ? ? ? ?{cr,<0.696.0>,#Ref<0.0.0.11414>,
? ? ? ? ? ? ? ? ? ? ? ?{[],[]},? ? ? ? ? ? ? ? ? ? ? ?2,? ? ? ? ? ? ? ? ?
? ? ?{[],[]},? ? ? ? ? ? ? ? ? ? ? ?{qstate,<0.695.0>,active,{0,nil}},? ? ?
? ? ? ? ? ? ? ? ?24}},? ? ? ? ? ? ? ? ? {{ch,<0.564.0>},
? ? ? ? ? ? ? ? ? ?{cr,<0.564.0>,#Ref<0.0.0.7285>,? ? ? ? ? ? ? ? ? ? ?
?{[28718,28699,28680,28661,28642],[28623]},? ? ? ? ? ? ? ? ? ? ? ?1,? ? ? ?
? ? ? ? ? ? ? ?{[],[]},? ? ? ? ? ? ? ? ? ? ? ?{qstate,<0.563.0>,active,
{0,nil}},
? ? ? ? ? ? ? ? ? ? ? ?12}},? ? ? ? ? ? ? ? ? {{ch,<0.644.0>},? ? ? ? ? ?
? ? ? ?{cr,<0.644.0>,#Ref<0.0.0.10071>,? ? ? ? ? ? ? ? ? ? ?
?{[28713,28694,28675,28656,28637,28618,28600,28582,? ? ? ? ? ? ? ? ? ? ? ?
?28564,28546,28528,28510,28492,28474,28456,28438,
? ? ? ? ? ? ? ? ? ? ? ? ?28420,28402,28384,28366,28348,28330,28311,28290,?
? ? ? ? ? ? ? ? ? ? ? ?28269,28248,28227,28206,28185,28164,28141,28118,? ? ?
? ? ? ? ? ? ? ? ? ?28095,28072,28049,28026,28003,27980,27957,27934,
? ? ? ? ? ? ? ? ? ? ? ? ?27911,27888,27865,27842,27819,27796,27773,27750,?
? ? ? ? ? ? ? ? ? ? ? ?27727,27704,27681,27658,27635,27612],? ? ? ? ? ? ? ?
? ? ? ? [27589]},? ? ? ? ? ? ? ? ? ? ? ?1,? ? ? ? ? ? ? ? ? ? ? ?{[],[]},
? ? ? ? ? ? ? ? ? ? ? ?{qstate,<0.643.0>,active,{0,nil}},? ? ? ? ? ? ? ? ?
? ? ?11}},? ? ? ? ? ? ? ? ? {{ch,<0.676.0>},? ? ? ? ? ? ? ? ? ?{cr,
<0.676.0>,#Ref<0.0.0.11309>,? ? ? ? ? ? ? ? ? ? ?
?{[28614,28613,28596,28595,28578,28577,28560,28559,
? ? ? ? ? ? ? ? ? ? ? ? ?28542,28541,28524,28523,28506,28505,28488,28487,?
? ? ? ? ? ? ? ? ? ? ? ?28470,28469,28452,28451,28434,28433,28416,28415,? ? ?
? ? ? ? ? ? ? ? ? ?28398,28397,28380,28379,28362,28361,28344,28343,
? ? ? ? ? ? ? ? ? ? ? ? ?28326,28325,28307,28306,28286,28285,28265,28264,?
? ? ? ? ? ? ? ? ? ? ? ?28244,28243,28223,28222,28202,28201,28181,28180,? ? ?
? ? ? ? ? ? ? ? ? ?28160,28159,28137,28136,28114,28113,28091,28090,
? ? ? ? ? ? ? ? ? ? ? ? ?28068,28067,28045,28044,28022,28021,27999,27998,?
? ? ? ? ? ? ? ? ? ? ? ?27976,27975,27953,27952,27930,27929,27907,27906,? ? ?
? ? ? ? ? ? ? ? ? ?27884,27883,27861,27860,27838,27837,27815,27814,
? ? ? ? ? ? ? ? ? ? ? ? ?27792,27791,27769,27768,27746,27745,27723,27722,?
? ? ? ? ? ? ? ? ? ? ? ?27700,27699,27677,27676,27654,27653,27631,27630,? ? ?
? ? ? ? ? ? ? ? ? ?27608,27607,27585],? ? ? ? ? ? ? ? ? ? ? ? [27584]},
? ? ? ? ? ? ? ? ? ? ? ?2,? ? ? ? ? ? ? ? ? ? ? ?{[{<0.676.0>,? ? ? ? ? ? ?
? ? ? ? ? ? {consumer,<<"amq.ctag-0cvDGmyaAbEW_j15AfhGHQ">>,? ? ? ? ? ? ? ?
? ? ? ? ? ? ? ? ? ? true}}],
? ? ? ? ? ? ? ? ? ? ? ? [{<0.676.0>,? ? ? ? ? ? ? ? ? ? ? ? ? {consumer,
<<"amq.ctag-YqU04SIfQv6P-QVueEjbRg">>,? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
true}}]},? ? ? ? ? ? ? ? ? ? ? ?{qstate,<0.675.0>,suspended,{0,nil}},
? ? ? ? ? ? ? ? ? ? ? ?25}},? ? ? ? ? ? ? ? ?
{{xtype_to_module,direct},rabbit_exchange_type_direct},? ? ? ? ? ? ? ? ?
{{credit_to,<7350.833.0>},31},? ? ? ? ? ? ? ? ? {{ch,<0.745.0>},? ? ? ? ? ?
? ? ? ?{cr,<0.745.0>,#Ref<0.0.0.12674>,
? ? ? ? ? ? ? ? ? ? ? ?{[28716,28697,28678,28659,28640,28621,28603,28585,?
? ? ? ? ? ? ? ? ? ? ? ?28567,28549,28531,28513,28495,28477,28459,28441,? ? ?
? ? ? ? ? ? ? ? ? ?28423,28405,28387,28369,28351,28333,28314,28293,
? ? ? ? ? ? ? ? ? ? ? ? ?28272,28251,28230,28209,28188,28167,28144,28121,?
? ? ? ? ? ? ? ? ? ? ? ?28098,28075,28052,28029,28006,27983,27960,27937,? ? ?
? ? ? ? ? ? ? ? ? ?27914,27891,27868,27845,27822,27799,27776,27753,
? ? ? ? ? ? ? ? ? ? ? ? ?27730,27707,27684,27661,27638,27615],? ? ? ? ? ?
? ? ? ? ? ? [27592]},? ? ? ? ? ? ? ? ? ? ? ?1,? ? ? ? ? ? ? ? ? ? ? ?{[],
[]},? ? ? ? ? ? ? ? ? ? ? ?{qstate,<0.744.0>,active,{0,nil}},
? ? ? ? ? ? ? ? ? ? ? ?37}},? ? ? ? ? ? ? ? ? {{ch,<0.660.0>},? ? ? ? ? ?
? ? ? ?{cr,<0.660.0>,#Ref<0.0.0.10650>,? ? ? ? ? ? ? ? ? ? ?
?{[28712,28693,28674,28655,28636,28617,28599,28581,? ? ? ? ? ? ? ? ? ? ? ?
?28563,28545,28527,28509,28491,28473,28455,28437,
? ? ? ? ? ? ? ? ? ? ? ? ?28419,28401,28383,28365,28347,28329,28310,28289,?
? ? ? ? ? ? ? ? ? ? ? ?28268,28247,28226,28205,28184,28163,28140,28117,? ? ?
? ? ? ? ? ? ? ? ? ?28094,28071,28048,28025,28002,27979,27956,27933,
? ? ? ? ? ? ? ? ? ? ? ? ?27910,27887,27864,27841,27818,27795,27772,27749,?
? ? ? ? ? ? ? ? ? ? ? ?27726,27703,27680,27657,27634,27611],? ? ? ? ? ? ? ?
? ? ? ? [27588]},? ? ? ? ? ? ? ? ? ? ? ?1,? ? ? ? ? ? ? ? ? ? ? ?{[],[]},
? ? ? ? ? ? ? ? ? ? ? ?{qstate,<0.659.0>,active,{0,nil}},? ? ? ? ? ? ? ? ?
? ? ?11}},? ? ? ? ? ? ? ? ? {{credit_to,<7350.837.0>},32},? ? ? ? ? ? ? ? ?
{{credit_to,<7350.881.0>},17},? ? ? ? ? ? ? ? ? {{ch,<0.737.0>},
? ? ? ? ? ? ? ? ? ?{cr,<0.737.0>,#Ref<0.0.0.12429>,? ? ? ? ? ? ? ? ? ? ?
?{[28612,28611,28594,28593,28576,28575,28558,28557,? ? ? ? ? ? ? ? ? ? ? ?
?28540,28539,28522,28521,28504,28503,28486,28485,
? ? ? ? ? ? ? ? ? ? ? ? ?28468,28467,28450,28449,28432,28431,28414,28413,?
? ? ? ? ? ? ? ? ? ? ? ?28396,28395,28378,28377,28360,28359,28342,28341,? ? ?
? ? ? ? ? ? ? ? ? ?28324,28323,28305,28304,28284,28283,28263,28262,
? ? ? ? ? ? ? ? ? ? ? ? ?28242,28241,28221,28220,28200,28199,28179,28178,?
? ? ? ? ? ? ? ? ? ? ? ?28158,28157,28135,28134,28112,28111,28089,28088,? ? ?
? ? ? ? ? ? ? ? ? ?28066,28065,28043,28042,28020,28019,27997,27996,
? ? ? ? ? ? ? ? ? ? ? ? ?27974,27973,27951,27950,27928,27927,27905,27904,?
? ? ? ? ? ? ? ? ? ? ? ?27882,27881,27859,27858,27836,27835,27813,27812,? ? ?
? ? ? ? ? ? ? ? ? ?27790,27789,27767,27766,27744,27743,27721,27720,
? ? ? ? ? ? ? ? ? ? ? ? ?27698,27697,27675,27674,27652,27651,27629,27628,?
? ? ? ? ? ? ? ? ? ? ? ?27606,27605,27583],? ? ? ? ? ? ? ? ? ? ? ? [27582]},?
? ? ? ? ? ? ? ? ? ? ?2,? ? ? ? ? ? ? ? ? ? ? ?{[{<0.737.0>,
? ? ? ? ? ? ? ? ? ? ? ? ? {consumer,<<"amq.ctag-
ITZqhulzHhUzx1uy1Eot3g">>,? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? true}}],? ? ?
? ? ? ? ? ? ? ? ? [{<0.737.0>,? ? ? ? ? ? ? ? ? ? ? ? ? {consumer,
<<"amq.ctag-hpo5ejdoJRdLrT6JS42jGw">>,
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? true}}]},? ? ? ? ? ? ? ? ? ? ?
?{qstate,<0.736.0>,suspended,{0,nil}},? ? ? ? ? ? ? ? ? ? ? ?11}},? ? ? ? ?
? ? ? ? {credit_blocked,[]},? ? ? ? ? ? ? ? ? {{credit_to,<7350.854.0>},3},
? ? ? ? ? ? ? ? ? {{ch,<0.716.0>},? ? ? ? ? ? ? ? ? ?{cr,
<0.716.0>,#Ref<0.0.0.11820>,? ? ? ? ? ? ? ? ? ? ?
?{[28610,28609,28592,28591,28574,28573,28556,28555,? ? ? ? ? ? ? ? ? ? ? ?
?28538,28537,28520,28519,28502,28501,28484,28483,
? ? ? ? ? ? ? ? ? ? ? ? ?28466,28465,28448,28447,28430,28429,28412,28411,?
? ? ? ? ? ? ? ? ? ? ? ?28394,28393,28376,28375,28358,28357,28340,28339,? ? ?
? ? ? ? ? ? ? ? ? ?28321,28320,28300,28299,28279,28278,28258,28257,
? ? ? ? ? ? ? ? ? ? ? ? ?28237,28236,28216,28215,28195,28194,28174,28173,?
? ? ? ? ? ? ? ? ? ? ? ?28153,28152,28130,28129,28107,28106,28084,28083,? ? ?
? ? ? ? ? ? ? ? ? ?28061,28060,28038,28037,28015,28014,27992,27991,
? ? ? ? ? ? ? ? ? ? ? ? ?27969,27968,27946,27945,27923,27922,27900,27899,?
? ? ? ? ? ? ? ? ? ? ? ?27877,27876,27854,27853,27831,27830,27808,27807,? ? ?
? ? ? ? ? ? ? ? ? ?27785,27784,27762,27761,27739,27738,27716,27715,
? ? ? ? ? ? ? ? ? ? ? ? ?27693,27692,27670,27669,27647,27646,27624,27623,?
? ? ? ? ? ? ? ? ? ? ? ?27601,27600,27578],? ? ? ? ? ? ? ? ? ? ? ? [27577]},?
? ? ? ? ? ? ? ? ? ? ?2,? ? ? ? ? ? ? ? ? ? ? ?{[{<0.716.0>,
? ? ? ? ? ? ? ? ? ? ? ? ? {consumer,<<"amq.ctag-BTlS-GmbXNpcU-
wUlMI60Q">>,? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? true}}],? ? ? ? ? ? ? ? ? ?
? ? [{<0.716.0>,? ? ? ? ? ? ? ? ? ? ? ? ? {consumer,<<"amq.ctag-
loc5monF8M1pz_Bb8ypumw">>,
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? true}}]},? ? ? ? ? ? ? ? ? ? ?
?{qstate,<0.715.0>,suspended,{0,nil}},? ? ? ? ? ? ? ? ? ? ? ?44}},? ? ? ? ?
? ? ? ? {{ch,<0.782.0>},? ? ? ? ? ? ? ? ? ?{cr,<0.782.0>,#Ref<0.0.0.15649>,
? ? ? ? ? ? ? ? ? ? ? ?{[28714,28695,28676,28657,28638,28619,28601,28583,?
? ? ? ? ? ? ? ? ? ? ? ?28565,28547,28529,28511,28493,28475,28457,28439,? ? ?
? ? ? ? ? ? ? ? ? ?28421,28403,28385,28367,28349,28331,28312,28291,
? ? ? ? ? ? ? ? ? ? ? ? ?28270,28249,28228,28207,28186,28165,28142,28119,?
? ? ? ? ? ? ? ? ? ? ? ?28096,28073,28050,28027,28004,27981,27958,27935,? ? ?
? ? ? ? ? ? ? ? ? ?27912,27889,27866,27843,27820,27797,27774,27751,
? ? ? ? ? ? ? ? ? ? ? ? ?27728,27705,27682,27659,27636,27613],? ? ? ? ? ?
? ? ? ? ? ? [27590]},? ? ? ? ? ? ? ? ? ? ? ?1,? ? ? ? ? ? ? ? ? ? ? ?{[],
[]},? ? ? ? ? ? ? ? ? ? ? ?{qstate,<0.781.0>,active,{0,nil}},
? ? ? ? ? ? ? ? ? ? ? ?35}},? ? ? ? ? ? ? ? ? {{credit_to,
<7350.846.0>},22},? ? ? ? ? ? ? ? ? {{ch,<0.684.0>},? ? ? ? ? ? ? ? ? ?{cr,
<0.684.0>,#Ref<0.0.0.11198>,? ? ? ? ? ? ? ? ? ? ?
?{[28711,28692,28673,28654,28635,28616,28598,28580,
? ? ? ? ? ? ? ? ? ? ? ? ?28562,28544,28526,28508,28490,28472,28454,28436,?
? ? ? ? ? ? ? ? ? ? ? ?28418,28400,28382,28364,28346,28328,28309,28288,? ? ?
? ? ? ? ? ? ? ? ? ?28267,28246,28225,28204,28183,28162,28139,28116,
? ? ? ? ? ? ? ? ? ? ? ? ?28093,28070,28047,28024,28001,27978,27955,27932,?
? ? ? ? ? ? ? ? ? ? ? ?27909,27886,27863,27840,27817,27794,27771,27748,? ? ?
? ? ? ? ? ? ? ? ? ?27725,27702,27679,27656,27633,27610],? ? ? ? ? ? ? ? ? ?
? ? [27587]},
? ? ? ? ? ? ? ? ? ? ? ?1,? ? ? ? ? ? ? ? ? ? ? ?{[],[]},? ? ? ? ? ? ? ? ?
? ? ?{qstate,<0.683.0>,active,{0,nil}},? ? ? ? ? ? ? ? ? ? ? ?11}},? ? ? ? ?
? ? ? ? {{ch,<0.692.0>},? ? ? ? ? ? ? ? ? ?{cr,<0.692.0>,#Ref<0.0.0.11088>,
? ? ? ? ? ? ? ? ? ? ? ?{[28710,28691,28672,28653,28634,28615,28597,28579,?
? ? ? ? ? ? ? ? ? ? ? ?28561,28543,28525,28507,28489,28471,28453,28435,? ? ?
? ? ? ? ? ? ? ? ? ?28417,28399,28381,28363,28345,28327,28308,28287,
? ? ? ? ? ? ? ? ? ? ? ? ?28266,28245,28224,28203,28182,28161,28138,28115,?
? ? ? ? ? ? ? ? ? ? ? ?28092,28069,28046,28023,28000,27977,27954,27931,? ? ?
? ? ? ? ? ? ? ? ? ?27908,27885,27862,27839,27816,27793,27770,27747,
? ? ? ? ? ? ? ? ? ? ? ? ?27724,27701,27678,27655,27632,27609],? ? ? ? ? ?
? ? ? ? ? ? [27586]},? ? ? ? ? ? ? ? ? ? ? ?1,? ? ? ? ? ? ? ? ? ? ? ?{[],
[]},? ? ? ? ? ? ? ? ? ? ? ?{qstate,<0.691.0>,active,{0,nil}},
? ? ? ? ? ? ? ? ? ? ? ?12}},? ? ? ? ? ? ? ? ? {{ch,<0.648.0>},? ? ? ? ? ?
? ? ? ?{cr,<0.648.0>,#Ref<0.0.0.10283>,? ? ? ? ? ? ? ? ? ? ?
?{[28717,28698,28679,28660,28641],[28622]},? ? ? ? ? ? ? ? ? ? ? ?1,
? ? ? ? ? ? ? ? ? ? ? ?{[],[]},? ? ? ? ? ? ? ? ? ? ? ?{qstate,
<0.647.0>,active,{0,nil}},? ? ? ? ? ? ? ? ? ? ? ?12}},? ? ? ? ? ? ? ? ?
{guid,{{2296174761,591959305,1562876388,1822892473},1}},
? ? ? ? ? ? ? ? ? {{ch,<0.720.0>},? ? ? ? ? ? ? ? ? ?{cr,
<0.720.0>,#Ref<0.0.0.12036>,? ? ? ? ? ? ? ? ? ? ?
?{[28723,28704,28685,28666,28647],[28628]},? ? ? ? ? ? ? ? ? ? ? ?1,? ? ? ?
? ? ? ? ? ? ? ?{[],[]},
? ? ? ? ? ? ? ? ? ? ? ?{qstate,<0.719.0>,active,{0,nil}},? ? ? ? ? ? ? ? ?
? ? ?37}},? ? ? ? ? ? ? ? ? {{credit_to,<7350.869.0>},39},? ? ? ? ? ? ? ? ?
{{ch,<0.636.0>},? ? ? ? ? ? ? ? ? ?{cr,<0.636.0>,#Ref<0.0.0.9960>,
? ? ? ? ? ? ? ? ? ? ? ?{[28705,28686,28667,28648,28629,28604,28586,28568,?
? ? ? ? ? ? ? ? ? ? ? ?28550,28532,28514,28496,28478,28460,28442,28424,? ? ?
? ? ? ? ? ? ? ? ? ?28406,28388,28370,28352,28334,28315,28294,28273,
? ? ? ? ? ? ? ? ? ? ? ? ?28252,28231,28210,28189,28168,28147,28124,28101,?
? ? ? ? ? ? ? ? ? ? ? ?28078,28055,28032,28009,27986,27963,27940,27917,? ? ?
? ? ? ? ? ? ? ? ? ?27894,27871,27848,27825,27802,27779,27756,27733,
? ? ? ? ? ? ? ? ? ? ? ? ?27710,27687,27664,27641,27618,27595],? ? ? ? ? ?
? ? ? ? ? ? [27572]},? ? ? ? ? ? ? ? ? ? ? ?1,? ? ? ? ? ? ? ? ? ? ? ?{[],
[]},? ? ? ? ? ? ? ? ? ? ? ?{qstate,<0.635.0>,active,{0,nil}},
? ? ? ? ? ? ? ? ? ? ? ?15}},? ? ? ? ? ? ? ? ? {{credit_to,
<7350.850.0>},50},? ? ? ? ? ? ? ? ? {{ch,<0.616.0>},? ? ? ? ? ? ? ? ? ?{cr,
<0.616.0>,#Ref<0.0.0.9300>,? ? ? ? ? ? ? ? ? ? ?
?{[28719,28700,28681,28662,28643],[28624]},
? ? ? ? ? ? ? ? ? ? ? ?1,? ? ? ? ? ? ? ? ? ? ? ?{[],[]},? ? ? ? ? ? ? ? ?
? ? ?{qstate,<0.615.0>,active,{0,nil}},? ? ? ? ? ? ? ? ? ? ? ?12}},? ? ? ? ?
? ? ? ? {{ch,<0.509.0>},? ? ? ? ? ? ? ? ? ?{cr,<0.509.0>,#Ref<0.0.0.2243>,
? ? ? ? ? ? ? ? ? ? ? ?{[28302,28281,28260,28239,28218,28197,28176,28155,?
? ? ? ? ? ? ? ? ? ? ? ?28132,28109,28086,28063,28040,28017,27994,27971],? ?
? ? ? ? ? ? ? ? ? ? [27580,27603,27626,27649,27672,27695,27718,27741,
? ? ? ? ? ? ? ? ? ? ? ? ?27764,27787,27810,27833,27856,27879,27902,27925,?
? ? ? ? ? ? ? ? ? ? ? ?27948]},? ? ? ? ? ? ? ? ? ? ? ?3,? ? ? ? ? ? ? ? ? ?
? ?{[],[]},? ? ? ? ? ? ? ? ? ? ? ?{qstate,<0.508.0>,active,{0,nil}},
? ? ? ? ? ? ? ? ? ? ? ?36}},? ? ? ? ? ? ? ? ? {{credit_to,
<7350.873.0>},17},? ? ? ? ? ? ? ? ? {{ch,<0.588.0>},? ? ? ? ? ? ? ? ? ?{cr,
<0.588.0>,#Ref<0.0.0.7808>,? ? ? ? ? ? ? ? ? ? ?
?{[28721,28702,28683,28664,28645],[28626]},
? ? ? ? ? ? ? ? ? ? ? ?1,? ? ? ? ? ? ? ? ? ? ? ?{[],[]},? ? ? ? ? ? ? ? ?
? ? ?{qstate,<0.587.0>,active,{0,nil}},? ? ? ? ? ? ? ? ? ? ? ?12}},? ? ? ? ?
? ? ? ? {{ch,<0.664.0>},? ? ? ? ? ? ? ? ? ?{cr,<0.664.0>,#Ref<0.0.0.10772>,
? ? ? ? ? ? ? ? ? ? ? ?{[28720,28701,28682,28663,28644],[28625]},? ? ? ? ?
? ? ? ? ? ? ?1,? ? ? ? ? ? ? ? ? ? ? ?{[],[]},? ? ? ? ? ? ? ? ? ? ?
?{qstate,<0.663.0>,active,{0,nil}},? ? ? ? ? ? ? ? ? ? ? ?13}},
? ? ? ? ? ? ? ? ? {fhc_age_tree,{0,nil}},? ? ? ? ? ? ? ? ? {{ch,
<0.708.0>},? ? ? ? ? ? ? ? ? ?{cr,<0.708.0>,#Ref<0.0.0.11695>,? ? ? ? ? ? ?
? ? ? ? ?{[28707,28688,28669,28650,28631,28606,28588,28570,
? ? ? ? ? ? ? ? ? ? ? ? ?28552,28534,28516,28498,28480,28462,28444,28426,?
? ? ? ? ? ? ? ? ? ? ? ?28408,28390,28372,28354,28336,28317,28296,28275,? ? ?
? ? ? ? ? ? ? ? ? ?28254,28233,28212,28191,28170,28149,28126,28103,
? ? ? ? ? ? ? ? ? ? ? ? ?28080,28057,28034,28011,27988,27965,27942,27919,?
? ? ? ? ? ? ? ? ? ? ? ?27896,27873,27850,27827,27804,27781,27758,27735,? ? ?
? ? ? ? ? ? ? ? ? ?27712,27689,27666,27643,27620,27597],? ? ? ? ? ? ? ? ? ?
? ? [27574]},
? ? ? ? ? ? ? ? ? ? ? ?1,? ? ? ? ? ? ? ? ? ? ? ?{[],[]},? ? ? ? ? ? ? ? ?
? ? ?{qstate,<0.707.0>,active,{0,nil}},? ? ? ? ? ? ? ? ? ? ? ?35}},? ? ? ? ?
? ? ? ? {{credit_to,<7350.829.0>},13},
? ? ? ? ? ? ? ? ? {{ch,<0.757.0>},? ? ? ? ? ? ? ? ? ?{cr,
<0.757.0>,#Ref<0.0.0.13150>,? ? ? ? ? ? ? ? ? ? ?
?{[28706,28687,28668,28649,28630,28605,28587,28569,? ? ? ? ? ? ? ? ? ? ? ?
?28551,28533,28515,28497,28479,28461,28443,28425,
? ? ? ? ? ? ? ? ? ? ? ? ?28407,28389,28371,28353,28335,28316,28295,28274,?
? ? ? ? ? ? ? ? ? ? ? ?28253,28232,28211,28190,28169,28148,28125,28102,? ? ?
? ? ? ? ? ? ? ? ? ?28079,28056,28033,28010,27987,27964,27941,27918,
? ? ? ? ? ? ? ? ? ? ? ? ?27895,27872,27849,27826,27803,27780,27757,27734,?
? ? ? ? ? ? ? ? ? ? ? ?27711,27688,27665,27642,27619,27596],? ? ? ? ? ? ? ?
? ? ? ? [27573]},? ? ? ? ? ? ? ? ? ? ? ?1,? ? ? ? ? ? ? ? ? ? ? ?{[],[]},
? ? ? ? ? ? ? ? ? ? ? ?{qstate,<0.756.0>,active,{0,nil}},? ? ? ? ? ? ? ? ?
? ? ?49}},? ? ? ? ? ? ? ? ? {{ch,<0.572.0>},? ? ? ? ? ? ? ? ? ?{cr,
<0.572.0>,#Ref<0.0.0.7536>,? ? ? ? ? ? ? ? ? ? ?
?{[28709,28690,28671,28652,28633,28608,28590,28572,
? ? ? ? ? ? ? ? ? ? ? ? ?28554,28536,28518,28500,28482,28464,28446,28428,?
? ? ? ? ? ? ? ? ? ? ? ?28410,28392,28374,28356,28338,28319,28298,28277,? ? ?
? ? ? ? ? ? ? ? ? ?28256,28235,28214,28193,28172,28151,28128,28105,
? ? ? ? ? ? ? ? ? ? ? ? ?28082,28059,28036,28013,27990,27967,27944,27921,?
? ? ? ? ? ? ? ? ? ? ? ?27898,27875,27852,27829,27806,27783,27760,27737,? ? ?
? ? ? ? ? ? ? ? ? ?27714,27691,27668,27645,27622,27599],? ? ? ? ? ? ? ? ? ?
? ? [27576]},
? ? ? ? ? ? ? ? ? ? ? ?1,? ? ? ? ? ? ? ? ? ? ? ?{[],[]},? ? ? ? ? ? ? ? ?
? ? ?{qstate,<0.571.0>,active,{0,nil}},? ? ? ? ? ? ? ? ? ? ? ?35}},? ? ? ? ?
? ? ? ? {{credit_to,<7350.825.0>},2},? ? ? ? ? ? ? ? ? {{credit_to,
<7350.858.0>},3}]
? ? trap_exit: true? ? status: running? ? heap_size: 121536? ? stack_size:
27? ? reductions: 9363545? neighbours:=SUPERVISOR REPORT==== 18-May-
2013::01:41:51 ===? ? ?Supervisor: {local,rabbit_amqqueue_sup}
? ? ?Context: ? ?child_terminated? ? ?Reason: ? ? {{case_clause,{empty,
{[],[]}}},? ? ? ? ? ? ? ? ? [{rabbit_amqqueue_process,subtract_acks,3,[]},?
? ? ? ? ? ? ? ? ?{rabbit_amqqueue_process,subtract_acks,4,[]},
? ? ? ? ? ? ? ? ? ?{rabbit_amqqueue_process,handle_cast,2,[]},? ? ? ? ? ?
? ? ? ?{gen_server2,handle_msg,2,[]},? ? ? ? ? ? ? ? ? ?{proc_lib,wake_up,3,
[{file,"proc_lib.erl"},{line,249}]}]}? ? ?Offender: ? [{pid,<0.301.0>},
? ? ? ? ? ? ? ? ? {name,rabbit_amqqueue},? ? ? ? ? ? ? ? ? {mfa,? ? ? ? ?
? ? ? ? ? ? {rabbit_amqqueue_process,start_link,? ? ? ? ? ? ? ? ? ? ? ? ?
[{amqqueue,? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?{resource,<<"/">>,queue,
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?<<"document_queue">>},? ? ? ? ? ? ? ? ?
? ? ? ? ? ? ?true,false,none,[],<0.7622.0>,[],[],undefined,? ? ? ? ? ? ? ? ?
? ? ? ? ? ? ?[]}]}},? ? ? ? ? ? ? ? ? {restart_type,temporary},
? ? ? ? ? ? ? ? ? {shutdown,4294967295},? ? ? ? ? ? ? ? ?
{child_type,worker}]
Notice how the case_clause is empty. ?The other rabbit log looks similar:

# cat /var/log/rabbitmq/rabbit-
72dRMgatfUn5amjFlPYPZgfGRAsWV3w2 at public.gmane.org?
?

=ERROR REPORT==== 18-May-2013::01:41:48 ===
** Generic server <0.301.0> terminating
** Last message in was {'$gen_cast',

? ? ? ? ? ? ? ? ? ? ? ? ? ?{ack,
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[27585,27584,27607,27630,27653,27676,27699,
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 27722,27745,27768,27791,27814,27837,27860,
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 27883,27906,27929,27952,27975,27998,28021,
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 28044,28067,28090,28113,28136,28159,28180,
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 28201,28222,28243,28264,28285,28306,28325,
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 28343,28361,28379,28397,28415,28433,28451,
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 28469,28487,28505,28523,28541,28559,28577,
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 28595,28613],
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?<0.676.0>}}
** When Server state == {q,

? ? ? ? ? ? ? ? ? ? ? ? ?{amqqueue,
? ? ? ? ? ? ? ? ? ? ? ? ? {resource,<<"/">>,queue,
? ? ? ? ? ? ? ? ? ? ? ? ? ?<<"document_queue">>},
? ? ? ? ? ? ? ? ? ? ? ? ? true,false,none,[],<0.301.0>,[],[],undefined, []},
? ? ? ? ? ? ? ? ? ? ? ? ?none,true,rabbit_variable_queue,
? ? ? ? ? ? ? ? ? ? ? ? ?{vqstate,

? ? ? ? ? ? ? ? ? ? ? ? ? {0,{[],[]}},
? ? ? ? ? ? ? ? ? ? ? ? ? {0,{[],[]}},
? ? ? ? ? ? ? ? ? ? ? ? ? {delta,undefined,0,undefined},
? ? ? ? ? ? ? ? ? ? ? ? ? {0,{[],[]}},

? ? ? ? ? ? ? ? ? ? ? ? ? {0,{[],[]}},?
? ? ? ? ? ? ? ? ? ? ? ? ? ... [ thousands of lines of state ] ...?
?

** Reason for termination ==
** {{case_clause,{empty,{[],[]}}},
? ? [{rabbit_amqqueue_process,subtract_acks,3,[]},
? ? ?{rabbit_amqqueue_process,subtract_acks,4,[]},
? ? ?{rabbit_amqqueue_process,handle_cast,2,[]},
? ? ?{gen_server2,handle_msg,2,[]},

? ? ?{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]}



=ERROR REPORT==== 18-May-2013::01:41:51 ===
connection <0.430.0>, channel 22 - soft error:
{amqp_error,not_found,

? ? ? ? ? ? "no queue 'document_queue' in vhost '/'",

? ? ? ? ? ? 'queue.declare'}

=ERROR REPORT==== 18-May-2013::01:41:51 ===

connection <0.384.0>, channel 12 - soft error:

{amqp_error,not_found,
? ? ? ? ? ? "no queue 'document_queue' in vhost '/'",
? ? ? ? ? ? 'queue.declare'}



=ERROR REPORT==== 18-May-2013::01:41:51 ===
connection <0.384.0>, channel 15 - soft error:
{amqp_error,not_found,

? ? ? ? ? ? "no queue 'document_queue' in vhost '/'",

? ? ? ? ? ? 'queue.declare'}

=ERROR REPORT==== 18-May-2013::01:41:51 ===

connection <0.373.0>, channel 23 - soft error:

{amqp_error,not_found,
? ? ? ? ? ? "no queue 'document_queue' in vhost '/'",
? ? ? ? ? ? 'queue.declare'}?

?... [ repeated a bunch of times ] ...

I also see this in my /var/log/syslog?with regards to beam.smp:


# cat /var/log/syslogMay 17 22:46:23 ip-10-123-123-123 kernel:
[244800.176041] INFO: task beam.smp:18971 blocked for more than 120
seconds.May 17 22:46:23 ip-10-123-123-123 kernel: [244800.176053] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 17 22:46:23 ip-10-123-123-123 kernel: [244800.176060] beam.smp ? ? ?
?D ffff8801bfd93700 ? ? 0 18971 ?18962 0x00000000May 17 22:46:23 ip-10-123-
123-123 kernel: [244800.176065] ?ffff8801b11cdcb8 0000000000000282
0000000000000000 ffffffffffffffe0
May 17 22:46:23 ip-10-123-123-123 kernel: [244800.176070]
?ffff8801b11cdfd8 ffff8801b11cdfd8 ffff8801b11cdfd8 0000000000013700May 17
22:46:23 ip-10-123-123-123 kernel: [244800.176075] ?ffff8801b3180000
ffff8801b35244a0 00007f500e2079e0 ffff8801b146bb80
May 17 22:46:23 ip-10-123-123-123 kernel: [244800.176079] Call Trace:May
17 22:46:23 ip-10-123-123-123 kernel: [244800.176089] ?[<ffffffff8165434f>]
schedule+0x3f/0x60May 17 22:46:23 ip-10-123-123-123 kernel: [244800.176094]
?[<ffffffff8106aef5>] exit_mm+0x85/0x130
May 17 22:46:23 ip-10-123-123-123 kernel: [244800.176097]
?[<ffffffff8106b10e>] do_exit+0x16e/0x450May 17 22:46:23 ip-10-123-123-123
kernel: [244800.176102] ?[<ffffffff810797aa>] ? __dequeue_signal+0x6a/0xb0
May 17 22:46:23 ip-10-123-123-123 kernel: [244800.176105]
?[<ffffffff8106b594>] do_group_exit+0x44/0xa0May 17 22:46:23 ip-10-123-123-
123 kernel: [244800.176108] ?[<ffffffff8107c36c>]
get_signal_to_deliver+0x21c/0x420
May 17 22:46:23 ip-10-123-123-123 kernel: [244800.176113]
?[<ffffffff81014825>] do_signal+0x45/0x130May 17 22:46:23 ip-10-123-123-123
kernel: [244800.176116] ?[<ffffffff8105539d>] ? set_next_entity+0xad/0xd0
May 17 22:46:23 ip-10-123-123-123 kernel: [244800.176120]
?[<ffffffff81014ad5>] do_notify_resume+0x65/0x80May 17 22:46:23 ip-10-123-
123-123 kernel: [244800.176124] ?[<ffffffff8165ea90>]
int_signal+0x12/0x17... [ repeated 9 more times with same timestamp ] ...
There are no strange CPU spikes on my EC2 node based on AWS monitoring.
The only thing that seems to stand out is that my writes to my EBS volume
spike during the crash (~90 write OPs/sec, 4 MiB/sec write bandwidth,
~30ms/op write latency).
My team and I are completely stumped and would appreciate any insight you
might have to the crash reports. ?Is this a problem with writes hanging when
going against the EBS volume since EBS volumes are essentially network
mounted? ?Is there a way to configure rabbit to be more tolerant of these
delays? ?

For debugging purposes, below is the status output of one of the nodes
when it is healthy (so you can see the versions of everything i'm running):


# rabbitmqctl statusStatus of node 'rabbit <at> ip-10-123-123-123' ...
[{pid,28978},?{running_applications,? ? ?[{rabbitmq_management,"RabbitMQ
Management Console","3.1.0"},? ? ? {rabbitmq_web_dispatch,"RabbitMQ Web
Dispatcher","3.1.0"},
? ? ? {webmachine,"webmachine","1.9.1-rmq3.1.0-git52e62bc"},? ? ?
{mochiweb,"MochiMedia Web Server","2.3.1-rmq3.1.0-gitd541e9a"},? ? ?
{rabbitmq_management_agent,"RabbitMQ Management Agent","3.1.0"},
? ? ? {rabbit,"RabbitMQ","3.1.0"},? ? ? {os_mon,"CPO ?CXC 138
46","2.2.11"},? ? ? {inets,"INETS ?CXC 138 49","5.9.4"},? ? ?
{mnesia,"MNESIA ?CXC 138 12","4.8"},
? ? ? {amqp_client,"RabbitMQ AMQP Client","3.1.0"},? ? ? {xmerl,"XML
parser","1.3.3"},? ? ? {sasl,"SASL ?CXC 138 11","2.3.1"},? ? ? {stdlib,"ERTS
?CXC 138 10","1.19.1"},
? ? ? {kernel,"ERTS ?CXC 138 10","2.16.1"}]},?{os,
{unix,linux}},?{erlang_version,? ? ?"Erlang R16B (erts-5.10.1) [source-
05f1189] [64-bit] [smp:8:8] [async-threads:30] [hipe] [kernel-
poll:true]\n"},
?{memory,? ? ?[{total,60179048},? ? ? {connection_procs,5274808},? ? ?
{queue_procs,13107264},? ? ? {plugins,354272},? ? ? {other_proc,9741830},
? ? ? {mnesia,150456},? ? ? {mgmt_db,12248},? ? ? {msg_index,34768},? ? ?
{other_ets,1148248},? ? ? {binary,4905584},? ? ? {code,19565863},? ? ?
{atom,703377},? ? ? {other_system,5180330}]},
>
?{vm_memory_high_watermark,0.8},?{vm_memory_limit,5828434329},?{disk_free_li
mit,1000000000},?{disk_free,3953344512},?{file_descriptors,? ?
?[{total_limit,924},{total_used,6},{sockets_limit,829},{sockets_used,4}]},
?{processes,[{limit,1048576},
{used,348}]},?{run_queue,0},?{uptime,3796}]...done.
Note that I am using Erlang v16 from the Erlang Solutions repo (esl-erlang
package). ?I originally was using Erlang v14 from Ubuntu repos (erlang-
base), but decided to upgrade hoping it would resolve the issue (which it
did not). ?I have also tried running RabbitMQ 3.0.4 without success. ?This
issue seems to affect both versions.
Thanks,
Karl
_______________________________________________
rabbitmq-discuss mailing listrabbitmq-discuss <at>
lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-
bin/mailman/listinfo/rabbitmq-discuss





<div>
<div dir="ltr">
<div>Karl,<br><br>Wild speculation would be that messages have been
consumed from a queue but that queue (name = document_queue?) is for some
reason missing in action (gone, inaccessible, ...) when the consumer
subsequently tries to ack the message(s). <br><br>
</div>Brett<br><br>
</div>
<div class="gmail_extra">
<br><br><div class="gmail_quote">On Sat, May 18, 2013 at 3:20 PM, Karl
Rieb <span dir="ltr">&lt;<a href="mailto:karl.rieb at ..."
target="_blank">karl.rieb <at> gmail.com</a>&gt;</span> wrote:<br>
<blockquote class="gmail_quote">Hi,<div><br></div>
<div>I am running a RabbitMQ cluster of 4 nodes in AWS EC2 on c1.xlarge
instances using Ubuntu 12.04 LTS (kernel 3.2.0-38). &nbsp;Periodically two
of my nodes holding the largest queues will crash and require me to restart
the rabbitmq-server service. &nbsp;Looking at the logs, I'm at a loss as to
why the crash is occurring. &nbsp;I've have tried reproducing the crash
manually by flooding my nodes with messages, but that doesn't seem to
trigger the issue. &nbsp;Below is output from various RabbitMQ logs:</div>
<div><br></div>
<blockquote class="gmail_quote"># cat /var/log/rabbitmq/rabbit <at> ip-10-
123-123-123-sasl.log<br>
=CRASH REPORT==== 18-May-2013::01:41:51 ===<br>&nbsp; crasher:<br>&nbsp;
&nbsp; initial call: gen:init_it/6<br>&nbsp; &nbsp; pid: &lt;0.301.0&gt;
<br>&nbsp; &nbsp; registered_name: []<br>&nbsp; &nbsp; exception exit:
{{case_clause,{empty,{[],[]}}},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;[{rabbit_amqqueue_process,subtract_acks,3,
[]},<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; {rabbit_amqqueue_process,subtract_acks,4,[]},<br>&nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{rabbit_amqqueue_process,handle_cast,2,[]},<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; {gen_server2,handle_msg,2,
[]},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; {proc_lib,wake_up,3,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; [{file,"proc_lib.erl"},
{line,249}]}]}<br>&nbsp; &nbsp; &nbsp; in function
&nbsp;gen_server2:terminate/3<br>&nbsp; &nbsp; ancestors:
[rabbit_amqqueue_sup,rabbit_sup,&lt;0.207.0&gt;]<br>&nbsp; &nbsp; messages:
[{'$gen_cast',{resume,&lt;0.676.0&gt;}},<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{'$gen_cast',{ack,[27583,27606,27629,27652,27675,27698,<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;27721,27744,27767,27790,27813,27836,<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp;27859,27882,27905,27928,27951,27974,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;27997,28020,28043,28066,28089,28112,<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp;28135,28158,28179,28200,28221,28242,<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;28263,28284,28305,28324,28342,28360,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;28378,28396,28414,28432,28450,28468,<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp;28486,28504,28522,28540,28558,28576,<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;28594,28612],<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;0.737.0&gt;}},<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{'$gen_cast',{resume,&lt;0.737.0&gt;}},<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; {'$gen_cast',{ack,
[27608,27631,27654,27677,27700,27723,<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27746,27769,27792,27815,27838,27861,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;27884,27907,27930,27953,27976,27999,<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp;28022,28045,28068,28091,28114,28137,<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;28160,28181,28202,28223,28244,28265,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;28286,28307,28326,28344,28362,28380,<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp;28398,28416,28434,28452,28470,28488,<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;28506,28524,28542,28560,28578,28596,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;28614],
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;0.676.0&gt;}},
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{'$gen_cast',{ack,[27578,27601,27624,27647,27670,27693,<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;27716,27739,27762,27785,27808,27831,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;27854,27877,27900,27923,27946,27969,<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp;27992,28015,28038,28061,28084,28107,<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;28130,28153,28174,28195,28216,28237,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;28258,28279,28300,28321,28340,28358,<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp;28376,28394,28412,28430,28448,28466,<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;28484,28502,28520,28538,28556,28574,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;28592,28610],
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;0.716.0&gt;}},
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{'$gen_cast',{resume,&lt;0.716.0&gt;}},<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; {'$gen_cast',{ack,
[27582,27605,27628,27651,27674,27697,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;27720,27743,27766,27789,27812,27835,<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp;27858,27881,27904,27927,27950,27973,<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;27996,28019,28042,28065,28088,28111,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;28134,28157,28178,28199,28220,28241,<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp;28262,28283,28304,28323,28341,28359,<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;28377,28395,28413,28431,28449,28467,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;28485,28503,28521,28539,28557,28575,<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp;28593,28611],<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &lt;0.737.0&gt;}}]<br>&nbsp; &nbsp; links:
[&lt;0.300.0&gt;]<br>
&nbsp; &nbsp; dictionary: [{{ch,&lt;0.729.0&gt;},<br>&nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{cr,&lt;0.729.0&gt;,#Ref&lt;0.0.0.12296&gt;,<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{[28715,28696,28677,28658,28639,28620,28602,28584,<br>&nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;28566,28548,28530,28512,28494,28476,28458,28440,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28422,28404,28386,28368,28350,28332,28313,28292,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28271,28250,28229,28208,28187,28166,28143,28120,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28097,28074,28051,28028,28005,27982,27959,27936,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27913,27890,27867,27844,27821,27798,27775,27752,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27729,27706,27683,27660,27637,27614],<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
[27591]},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;1,<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;{[],[]},<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;{qstate,&lt;0.728.0&gt;,active,{0,nil}},<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;11}},
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{{credit_to,&lt;7350.877.0&gt;},22},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; {{ch,&lt;0.608.0&gt;},<br>&nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{cr,&lt;0.608.0&gt;,#Ref&lt;0.0.0.9041&gt;,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;{[28708,28689,28670,28651,28632,28607,28589,28571,<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;28553,28535,28517,28499,28481,28463,28445,28427,<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;28409,28391,28373,28355,28337,28318,28297,28276,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28255,28234,28213,28192,28171,28150,28127,28104,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28081,28058,28035,28012,27989,27966,27943,27920,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27897,27874,27851,27828,27805,27782,27759,27736,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27713,27690,27667,27644,27621,27598],<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
[27575]},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;1,<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;{[],[]},<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{qstate,&lt;0.607.0&gt;,active,{0,nil}},<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;38}},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; {{ch,&lt;0.624.0&gt;},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{cr,&lt;0.624.0&gt;,#Ref&lt;0.0.0.9436&gt;,<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{[28722,28703,28684,28665,28646],[28627]},<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;1,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;{[],[]},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;{qstate,&lt;0.623.0&gt;,active,{0,nil}},
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;36}},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; {{ch,&lt;0.696.0&gt;},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{cr,&lt;0.696.0&gt;,#Ref&lt;0.0.0.11414&gt;,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;{[],[]},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;2,<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;{[],[]},<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{qstate,&lt;0.695.0&gt;,active,{0,nil}},<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;24}},<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{{ch,&lt;0.564.0&gt;},<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{cr,&lt;0.564.0&gt;,#Ref&lt;0.0.0.7285&gt;,<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{[28718,28699,28680,28661,28642],[28623]},<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;1,<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;{[],
[]},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;{qstate,&lt;0.563.0&gt;,active,{0,nil}},<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;12}},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; {{ch,&lt;0.644.0&gt;},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{cr,&lt;0.644.0&gt;,#Ref&lt;0.0.0.10071&gt;,<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{[28713,28694,28675,28656,28637,28618,28600,28582,<br>&nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;28564,28546,28528,28510,28492,28474,28456,28438,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28420,28402,28384,28366,28348,28330,28311,28290,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28269,28248,28227,28206,28185,28164,28141,28118,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28095,28072,28049,28026,28003,27980,27957,27934,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27911,27888,27865,27842,27819,27796,27773,27750,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27727,27704,27681,27658,27635,27612],<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
[27589]},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;1,<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;{[],[]},<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;{qstate,&lt;0.643.0&gt;,active,{0,nil}},<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;11}},
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{{ch,&lt;0.676.0&gt;},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;{cr,&lt;0.676.0&gt;,#Ref&lt;0.0.0.11309&gt;,<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{[28614,28613,28596,28595,28578,28577,28560,28559,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28542,28541,28524,28523,28506,28505,28488,28487,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28470,28469,28452,28451,28434,28433,28416,28415,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28398,28397,28380,28379,28362,28361,28344,28343,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28326,28325,28307,28306,28286,28285,28265,28264,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28244,28243,28223,28222,28202,28201,28181,28180,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28160,28159,28137,28136,28114,28113,28091,28090,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28068,28067,28045,28044,28022,28021,27999,27998,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27976,27975,27953,27952,27930,27929,27907,27906,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27884,27883,27861,27860,27838,27837,27815,27814,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27792,27791,27769,27768,27746,27745,27723,27722,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27700,27699,27677,27676,27654,27653,27631,27630,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27608,27607,27585],<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; [27584]},<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;2,<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp;{[{&lt;0.676.0&gt;,<br>&nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{consumer,&lt;&lt;"amq.ctag-0cvDGmyaAbEW_j15AfhGHQ"&gt;&gt;,<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; true}}],<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; [{&lt;0.676.0&gt;,<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{consumer,&lt;&lt;"amq.ctag-YqU04SIfQv6P-QVueEjbRg"&gt;&gt;,<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; true}}]},<br>&nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{qstate,&lt;0.675.0&gt;,suspended,{0,nil}},<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;25}},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; {{xtype_to_module,direct},rabbit_exchange_type_direct},
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{{credit_to,&lt;7350.833.0&gt;},31},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; {{ch,&lt;0.745.0&gt;},<br>&nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{cr,&lt;0.745.0&gt;,#Ref&lt;0.0.0.12674&gt;,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;{[28716,28697,28678,28659,28640,28621,28603,28585,<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;28567,28549,28531,28513,28495,28477,28459,28441,<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;28423,28405,28387,28369,28351,28333,28314,28293,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28272,28251,28230,28209,28188,28167,28144,28121,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28098,28075,28052,28029,28006,27983,27960,27937,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27914,27891,27868,27845,27822,27799,27776,27753,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27730,27707,27684,27661,27638,27615],<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
[27592]},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;1,<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;{[],[]},<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{qstate,&lt;0.744.0&gt;,active,{0,nil}},<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;37}},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; {{ch,&lt;0.660.0&gt;},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{cr,&lt;0.660.0&gt;,#Ref&lt;0.0.0.10650&gt;,<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{[28712,28693,28674,28655,28636,28617,28599,28581,<br>&nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;28563,28545,28527,28509,28491,28473,28455,28437,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28419,28401,28383,28365,28347,28329,28310,28289,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28268,28247,28226,28205,28184,28163,28140,28117,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28094,28071,28048,28025,28002,27979,27956,27933,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27910,27887,27864,27841,27818,27795,27772,27749,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27726,27703,27680,27657,27634,27611],<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
[27588]},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;1,<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;{[],[]},<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;{qstate,&lt;0.659.0&gt;,active,{0,nil}},<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;11}},
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{{credit_to,&lt;7350.837.0&gt;},32},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; {{credit_to,&lt;7350.881.0&gt;},17},<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{{ch,&lt;0.737.0&gt;},<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{cr,&lt;0.737.0&gt;,#Ref&lt;0.0.0.12429&gt;,<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{[28612,28611,28594,28593,28576,28575,28558,28557,<br>&nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;28540,28539,28522,28521,28504,28503,28486,28485,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28468,28467,28450,28449,28432,28431,28414,28413,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28396,28395,28378,28377,28360,28359,28342,28341,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28324,28323,28305,28304,28284,28283,28263,28262,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28242,28241,28221,28220,28200,28199,28179,28178,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28158,28157,28135,28134,28112,28111,28089,28088,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28066,28065,28043,28042,28020,28019,27997,27996,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27974,27973,27951,27950,27928,27927,27905,27904,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27882,27881,27859,27858,27836,27835,27813,27812,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27790,27789,27767,27766,27744,27743,27721,27720,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27698,27697,27675,27674,27652,27651,27629,27628,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27606,27605,27583],<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; [27582]},<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;2,<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;{[{&lt;0.737.0&gt;,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; {consumer,&lt;&lt;"amq.ctag-
ITZqhulzHhUzx1uy1Eot3g"&gt;&gt;,<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; true}}],<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; [{&lt;0.737.0&gt;,<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{consumer,&lt;&lt;"amq.ctag-hpo5ejdoJRdLrT6JS42jGw"&gt;&gt;,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; true}}]},<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{qstate,&lt;0.736.0&gt;,suspended,{0,nil}},<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;11}},<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; {credit_blocked,[]},
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{{credit_to,&lt;7350.854.0&gt;},3},<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{{ch,&lt;0.716.0&gt;},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;{cr,&lt;0.716.0&gt;,#Ref&lt;0.0.0.11820&gt;,<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{[28610,28609,28592,28591,28574,28573,28556,28555,<br>&nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;28538,28537,28520,28519,28502,28501,28484,28483,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28466,28465,28448,28447,28430,28429,28412,28411,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28394,28393,28376,28375,28358,28357,28340,28339,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28321,28320,28300,28299,28279,28278,28258,28257,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28237,28236,28216,28215,28195,28194,28174,28173,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28153,28152,28130,28129,28107,28106,28084,28083,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28061,28060,28038,28037,28015,28014,27992,27991,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27969,27968,27946,27945,27923,27922,27900,27899,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27877,27876,27854,27853,27831,27830,27808,27807,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27785,27784,27762,27761,27739,27738,27716,27715,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27693,27692,27670,27669,27647,27646,27624,27623,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27601,27600,27578],<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; [27577]},<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;2,<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;{[{&lt;0.716.0&gt;,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; {consumer,&lt;&lt;"amq.ctag-BTlS-GmbXNpcU-
wUlMI60Q"&gt;&gt;,<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
true}}],<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; [{&lt;0.716.0&gt;,<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{consumer,&lt;&lt;"amq.ctag-loc5monF8M1pz_Bb8ypumw"&gt;&gt;,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; true}}]},<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{qstate,&lt;0.715.0&gt;,suspended,{0,nil}},<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;44}},<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{{ch,&lt;0.782.0&gt;},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;{cr,&lt;0.782.0&gt;,#Ref&lt;0.0.0.15649&gt;,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;{[28714,28695,28676,28657,28638,28619,28601,28583,<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;28565,28547,28529,28511,28493,28475,28457,28439,<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;28421,28403,28385,28367,28349,28331,28312,28291,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28270,28249,28228,28207,28186,28165,28142,28119,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28096,28073,28050,28027,28004,27981,27958,27935,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27912,27889,27866,27843,27820,27797,27774,27751,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27728,27705,27682,27659,27636,27613],<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
[27590]},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;1,<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;{[],[]},<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{qstate,&lt;0.781.0&gt;,active,{0,nil}},<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;35}},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; {{credit_to,&lt;7350.846.0&gt;},22},<br>&nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; {{ch,&lt;0.684.0&gt;},<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{cr,&lt;0.684.0&gt;,#Ref&lt;0.0.0.11198&gt;,<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{[28711,28692,28673,28654,28635,28616,28598,28580,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28562,28544,28526,28508,28490,28472,28454,28436,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28418,28400,28382,28364,28346,28328,28309,28288,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28267,28246,28225,28204,28183,28162,28139,28116,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28093,28070,28047,28024,28001,27978,27955,27932,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27909,27886,27863,27840,27817,27794,27771,27748,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27725,27702,27679,27656,27633,27610],<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
[27587]},<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;1,<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp;{[],[]},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{qstate,&lt;0.683.0&gt;,active,{0,nil}},<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;11}},<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{{ch,&lt;0.692.0&gt;},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;{cr,&lt;0.692.0&gt;,#Ref&lt;0.0.0.11088&gt;,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;{[28710,28691,28672,28653,28634,28615,28597,28579,<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;28561,28543,28525,28507,28489,28471,28453,28435,<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;28417,28399,28381,28363,28345,28327,28308,28287,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28266,28245,28224,28203,28182,28161,28138,28115,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28092,28069,28046,28023,28000,27977,27954,27931,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27908,27885,27862,27839,27816,27793,27770,27747,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27724,27701,27678,27655,27632,27609],<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
[27586]},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;1,<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;{[],[]},<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{qstate,&lt;0.691.0&gt;,active,{0,nil}},<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;12}},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; {{ch,&lt;0.648.0&gt;},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{cr,&lt;0.648.0&gt;,#Ref&lt;0.0.0.10283&gt;,<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{[28717,28698,28679,28660,28641],[28622]},<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;1,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;{[],[]},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;{qstate,&lt;0.647.0&gt;,active,{0,nil}},
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;12}},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; {guid,{{<a href="tel:2296174761" value="+12296174761"
target="_blank">2296174761</a>,591959305,1562876388,1822892473},1}},<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{{ch,&lt;0.720.0&gt;},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;{cr,&lt;0.720.0&gt;,#Ref&lt;0.0.0.12036&gt;,<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{[28723,28704,28685,28666,28647],[28628]},<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;1,<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;{[],
[]},<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;{qstate,&lt;0.719.0&gt;,active,{0,nil}},<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;37}},
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{{credit_to,&lt;7350.869.0&gt;},39},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; {{ch,&lt;0.636.0&gt;},<br>&nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{cr,&lt;0.636.0&gt;,#Ref&lt;0.0.0.9960&gt;,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;{[28705,28686,28667,28648,28629,28604,28586,28568,<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;28550,28532,28514,28496,28478,28460,28442,28424,<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;28406,28388,28370,28352,28334,28315,28294,28273,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28252,28231,28210,28189,28168,28147,28124,28101,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28078,28055,28032,28009,27986,27963,27940,27917,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27894,27871,27848,27825,27802,27779,27756,27733,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27710,27687,27664,27641,27618,27595],<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
[27572]},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;1,<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;{[],[]},<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{qstate,&lt;0.635.0&gt;,active,{0,nil}},<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;15}},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; {{credit_to,&lt;7350.850.0&gt;},50},<br>&nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; {{ch,&lt;0.616.0&gt;},<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{cr,&lt;0.616.0&gt;,#Ref&lt;0.0.0.9300&gt;,<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{[28719,28700,28681,28662,28643],[28624]},<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;1,<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp;{[],[]},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{qstate,&lt;0.615.0&gt;,active,{0,nil}},<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;12}},<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{{ch,&lt;0.509.0&gt;},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;{cr,&lt;0.509.0&gt;,#Ref&lt;0.0.0.2243&gt;,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;{[28302,28281,28260,28239,28218,28197,28176,28155,<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;28132,28109,28086,28063,28040,28017,27994,27971],<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
[27580,27603,27626,27649,27672,27695,27718,27741,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27764,27787,27810,27833,27856,27879,27902,27925,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27948]},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;3,<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;{[],[]},<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{qstate,&lt;0.508.0&gt;,active,{0,nil}},<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;36}},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; {{credit_to,&lt;7350.873.0&gt;},17},<br>&nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; {{ch,&lt;0.588.0&gt;},<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{cr,&lt;0.588.0&gt;,#Ref&lt;0.0.0.7808&gt;,<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{[28721,28702,28683,28664,28645],[28626]},<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;1,<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp;{[],[]},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{qstate,&lt;0.587.0&gt;,active,{0,nil}},<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;12}},<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{{ch,&lt;0.664.0&gt;},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;{cr,&lt;0.664.0&gt;,#Ref&lt;0.0.0.10772&gt;,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;{[28720,28701,28682,28663,28644],[28625]},<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;1,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;{[],[]},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;{qstate,&lt;0.663.0&gt;,active,{0,nil}},
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;13}},<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{fhc_age_tree,{0,nil}},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; {{ch,&lt;0.708.0&gt;},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{cr,&lt;0.708.0&gt;,#Ref&lt;0.0.0.11695&gt;,<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{[28707,28688,28669,28650,28631,28606,28588,28570,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28552,28534,28516,28498,28480,28462,28444,28426,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28408,28390,28372,28354,28336,28317,28296,28275,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28254,28233,28212,28191,28170,28149,28126,28103,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28080,28057,28034,28011,27988,27965,27942,27919,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27896,27873,27850,27827,27804,27781,27758,27735,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27712,27689,27666,27643,27620,27597],<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
[27574]},<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;1,<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp;{[],[]},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{qstate,&lt;0.707.0&gt;,active,{0,nil}},<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;35}},<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{{credit_to,&lt;7350.829.0&gt;},13},<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{{ch,&lt;0.757.0&gt;},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;{cr,&lt;0.757.0&gt;,#Ref&lt;0.0.0.13150&gt;,<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{[28706,28687,28668,28649,28630,28605,28587,28569,<br>&nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;28551,28533,28515,28497,28479,28461,28443,28425,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28407,28389,28371,28353,28335,28316,28295,28274,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28253,28232,28211,28190,28169,28148,28125,28102,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28079,28056,28033,28010,27987,27964,27941,27918,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27895,27872,27849,27826,27803,27780,27757,27734,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27711,27688,27665,27642,27619,27596],<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
[27573]},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;1,<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;{[],[]},<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;{qstate,&lt;0.756.0&gt;,active,{0,nil}},<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;49}},
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{{ch,&lt;0.572.0&gt;},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;{cr,&lt;0.572.0&gt;,#Ref&lt;0.0.0.7536&gt;,<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{[28709,28690,28671,28652,28633,28608,28590,28572,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28554,28536,28518,28500,28482,28464,28446,28428,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28410,28392,28374,28356,28338,28319,28298,28277,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28256,28235,28214,28193,28172,28151,28128,28105,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;28082,28059,28036,28013,27990,27967,27944,27921,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27898,27875,27852,27829,27806,27783,27760,27737,
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;27714,27691,27668,27645,27622,27599],<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
[27576]},<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;1,<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp;{[],[]},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{qstate,&lt;0.571.0&gt;,active,{0,nil}},<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;35}},<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{{credit_to,&lt;7350.825.0&gt;},2},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; {{credit_to,&lt;7350.858.0&gt;},3}]<br>
&nbsp; &nbsp; trap_exit: true<br>&nbsp; &nbsp; status: running<br>&nbsp;
&nbsp; heap_size: 121536<br>&nbsp; &nbsp; stack_size: 27<br>&nbsp; &nbsp;
reductions: 9363545<br>&nbsp; neighbours:<br><br>=SUPERVISOR REPORT==== 18-
May-2013::01:41:51 ===<br>&nbsp; &nbsp; &nbsp;Supervisor:
{local,rabbit_amqqueue_sup}<br>
&nbsp; &nbsp; &nbsp;Context: &nbsp; &nbsp;child_terminated<br>&nbsp;
&nbsp; &nbsp;Reason: &nbsp; &nbsp; {{case_clause,{empty,{[],[]}}},<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
[{rabbit_amqqueue_process,subtract_acks,3,[]},<br>&nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{rabbit_amqqueue_process,subtract_acks,4,[]},<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{rabbit_amqqueue_process,handle_cast,2,[]},<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;{gen_server2,handle_msg,2,[]},
<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]}<br>&nbsp; &nbsp;
&nbsp;Offender: &nbsp; [{pid,&lt;0.301.0&gt;},<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{name,rabbit_amqqueue},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; {mfa,<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; {rabbit_amqqueue_process,start_link,<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; [{amqqueue,<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{resource,&lt;&lt;"/"&gt;&gt;,queue,<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;&lt;&lt;"document_queue"&gt;&gt;},<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;true,false,none,[],&lt;0.7622.0&gt;,[],[],undefined,<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;[]}]}},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; {restart_type,temporary},<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{shutdown,4294967295},<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; {child_type,worker}]</blockquote>
<div><br></div>
<div>Notice how the case_clause is empty. &nbsp;The other rabbit log looks
similar:</div>
<div><br></div>
<blockquote class="gmail_quote"># cat /var/log/rabbitmq/rabbit at ...&nbsp;
</blockquote>
<blockquote class="gmail_quote">&nbsp;</blockquote>
<blockquote class="gmail_quote">
=ERROR REPORT==== 18-May-2013::01:41:48 ===</blockquote>
<blockquote class="gmail_quote">** Generic server &lt;0.301.0&gt;
terminating</blockquote>
<blockquote class="gmail_quote">** Last message in was {'$gen_cast',
</blockquote>
<blockquote class="gmail_quote">
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp;{ack,</blockquote>
<blockquote class="gmail_quote">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
[27585,27584,27607,27630,27653,27676,27699,</blockquote>
<blockquote class="gmail_quote">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
27722,27745,27768,27791,27814,27837,27860,</blockquote>
<blockquote class="gmail_quote">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
27883,27906,27929,27952,27975,27998,28021,</blockquote>
<blockquote class="gmail_quote">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
28044,28067,28090,28113,28136,28159,28180,</blockquote>
<blockquote class="gmail_quote">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
28201,28222,28243,28264,28285,28306,28325,</blockquote>
<blockquote class="gmail_quote">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
28343,28361,28379,28397,28415,28433,28451,</blockquote>
<blockquote class="gmail_quote">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
28469,28487,28505,28523,28541,28559,28577,</blockquote>
<blockquote class="gmail_quote">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
28595,28613],</blockquote>
<blockquote class="gmail_quote">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;&lt;0.676.0&gt;}}</blockquote>
<blockquote class="gmail_quote">** When Server state == {q,</blockquote>
<blockquote class="gmail_quote">
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;{amqqueue,</blockquote>
<blockquote class="gmail_quote">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{resource,&lt;&lt;"/"&gt;&gt;,queue,</blockquote>
<blockquote class="gmail_quote">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;&lt;&lt;"document_queue"&gt;&gt;},</blockquote>
<blockquote class="gmail_quote">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; true,false,none,
[],&lt;0.301.0&gt;,[],[],undefined,[]},</blockquote>
<blockquote class="gmail_quote">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;none,true,rabbit_variable_queue,</blockquote>
<blockquote class="gmail_quote">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;{vqstate,</blockquote>
<blockquote class="gmail_quote">
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; {0,{[],[]}},</blockquote>
<blockquote class="gmail_quote">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; {0,{[],[]}},</blockquote>
<blockquote class="gmail_quote">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
{delta,undefined,0,undefined},</blockquote>
<blockquote class="gmail_quote">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; {0,{[],[]}},</blockquote>
<blockquote class="gmail_quote">
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; {0,{[],[]}},&nbsp;</blockquote>
<blockquote class="gmail_quote">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ... [ thousands of lines of
state ] ...&nbsp;</blockquote>
<blockquote class="gmail_quote">&nbsp;</blockquote>
<blockquote class="gmail_quote">
** Reason for termination ==</blockquote>
<blockquote class="gmail_quote">** {{case_clause,{empty,{[],[]}}},
</blockquote>
<blockquote class="gmail_quote">&nbsp; &nbsp;
[{rabbit_amqqueue_process,subtract_acks,3,[]},</blockquote>
<blockquote class="gmail_quote">&nbsp; &nbsp; &nbsp;
{rabbit_amqqueue_process,subtract_acks,4,[]},</blockquote>
<blockquote class="gmail_quote">&nbsp; &nbsp; &nbsp;
{rabbit_amqqueue_process,handle_cast,2,[]},</blockquote>
<blockquote class="gmail_quote">&nbsp; &nbsp; &nbsp;
{gen_server2,handle_msg,2,[]},</blockquote>
<blockquote class="gmail_quote">
&nbsp; &nbsp; &nbsp;{proc_lib,wake_up,3,[{file,"proc_lib.erl"},
{line,249}]}]}</blockquote>
<blockquote class="gmail_quote">
<br>
</blockquote>
<blockquote class="gmail_quote">=ERROR REPORT==== 18-May-2013::01:41:51
===</blockquote>
<blockquote class="gmail_quote">connection &lt;0.430.0&gt;, channel 22 -
soft error:</blockquote>
<blockquote class="gmail_quote">{amqp_error,not_found,</blockquote>
<blockquote class="gmail_quote">
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "no queue 'document_queue' in
vhost '/'",</blockquote>
<blockquote class="gmail_quote">
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 'queue.declare'}</blockquote>
<blockquote class="gmail_quote"><br></blockquote>
<blockquote class="gmail_quote">=ERROR REPORT==== 18-May-2013::01:41:51
===</blockquote>
<blockquote class="gmail_quote">
connection &lt;0.384.0&gt;, channel 12 - soft error:</blockquote>
<blockquote class="gmail_quote">
{amqp_error,not_found,</blockquote>
<blockquote class="gmail_quote">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
"no queue 'document_queue' in vhost '/'",</blockquote>
<blockquote class="gmail_quote">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
'queue.declare'}</blockquote>
<blockquote class="gmail_quote">
<br>
</blockquote>
<blockquote class="gmail_quote">=ERROR REPORT==== 18-May-2013::01:41:51
===</blockquote>
<blockquote class="gmail_quote">connection &lt;0.384.0&gt;, channel 15 -
soft error:</blockquote>
<blockquote class="gmail_quote">{amqp_error,not_found,</blockquote>
<blockquote class="gmail_quote">
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "no queue 'document_queue' in
vhost '/'",</blockquote>
<blockquote class="gmail_quote">
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 'queue.declare'}</blockquote>
<blockquote class="gmail_quote"><br></blockquote>
<blockquote class="gmail_quote">=ERROR REPORT==== 18-May-2013::01:41:51
===</blockquote>
<blockquote class="gmail_quote">
connection &lt;0.373.0&gt;, channel 23 - soft error:</blockquote>
<blockquote class="gmail_quote">
{amqp_error,not_found,</blockquote>
<blockquote class="gmail_quote">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
"no queue 'document_queue' in vhost '/'",</blockquote>
<blockquote class="gmail_quote">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
'queue.declare'}&nbsp;</blockquote>
<blockquote class="gmail_quote">
&nbsp;... [ repeated a bunch of times ] ...</blockquote>
<div><br></div>
<div>I also see this in my /var/log/syslog&nbsp;with regards to beam.smp: </div>
<div><br></div>
<blockquote class="gmail_quote">
# cat /var/log/syslog<br>May 17 22:46:23 ip-10-123-123-123 kernel:
[244800.176041] INFO: task beam.smp:18971 blocked for more than 120 seconds.
<br>May 17 22:46:23 ip-10-123-123-123 kernel: [244800.176053] "echo 0 &gt;
/proc/sys/kernel/hung_task_timeout_secs" disables this message.<br>
May 17 22:46:23 ip-10-123-123-123 kernel: [244800.176060] beam.smp &nbsp;
&nbsp; &nbsp; &nbsp;D ffff8801bfd93700 &nbsp; &nbsp; 0 18971 &nbsp;18962
0x00000000<br>May 17 22:46:23 ip-10-123-123-123 kernel: [244800.176065]
&nbsp;ffff8801b11cdcb8 0000000000000282 0000000000000000
ffffffffffffffe0<br>
May 17 22:46:23 ip-10-123-123-123 kernel: [244800.176070]
&nbsp;ffff8801b11cdfd8 ffff8801b11cdfd8 ffff8801b11cdfd8
0000000000013700<br>May 17 22:46:23 ip-10-123-123-123 kernel:
[244800.176075] &nbsp;ffff8801b3180000 ffff8801b35244a0 00007f500e2079e0
ffff8801b146bb80<br>
May 17 22:46:23 ip-10-123-123-123 kernel: [244800.176079] Call Trace:
<br>May 17 22:46:23 ip-10-123-123-123 kernel: [244800.176089] &nbsp;
[&lt;ffffffff8165434f&gt;] schedule+0x3f/0x60<br>May 17 22:46:23 ip-10-123-
123-123 kernel: [244800.176094] &nbsp;[&lt;ffffffff8106aef5&gt;]
exit_mm+0x85/0x130<br>
May 17 22:46:23 ip-10-123-123-123 kernel: [244800.176097] &nbsp;
[&lt;ffffffff8106b10e&gt;] do_exit+0x16e/0x450<br>May 17 22:46:23 ip-10-123-
123-123 kernel: [244800.176102] &nbsp;[&lt;ffffffff810797aa&gt;] ?
__dequeue_signal+0x6a/0xb0<br>
May 17 22:46:23 ip-10-123-123-123 kernel: [244800.176105] &nbsp;
[&lt;ffffffff8106b594&gt;] do_group_exit+0x44/0xa0<br>May 17 22:46:23 ip-10-
123-123-123 kernel: [244800.176108] &nbsp;[&lt;ffffffff8107c36c&gt;]
get_signal_to_deliver+0x21c/0x420<br>
May 17 22:46:23 ip-10-123-123-123 kernel: [244800.176113] &nbsp;
[&lt;ffffffff81014825&gt;] do_signal+0x45/0x130<br>May 17 22:46:23 ip-10-
123-123-123 kernel: [244800.176116] &nbsp;[&lt;ffffffff8105539d&gt;] ?
set_next_entity+0xad/0xd0<br>
May 17 22:46:23 ip-10-123-123-123 kernel: [244800.176120] &nbsp;
[&lt;ffffffff81014ad5&gt;] do_notify_resume+0x65/0x80<br>May 17 22:46:23 ip-
10-123-123-123 kernel: [244800.176124] &nbsp;[&lt;ffffffff8165ea90&gt;]
int_signal+0x12/0x17<br>... [ repeated 9 more times with same timestamp ]
...</blockquote>
<div><br></div>
<div>There are no strange CPU spikes on my EC2 node based on AWS
monitoring. The only thing that seems to stand out is that my writes to my
EBS volume spike during the crash (~90 write OPs/sec, 4 MiB/sec write
bandwidth, ~30ms/op write latency).</div>
<div><br></div>
<div>My team and I are completely stumped and would appreciate any insight
you might have to the crash reports. &nbsp;Is this a problem with writes
hanging when going against the EBS volume since EBS volumes are essentially
network mounted? &nbsp;Is there a way to configure rabbit to be more
tolerant of these delays? &nbsp;</div>
<div><br></div>
<div><br></div>
<div>For debugging purposes, below is the status output of one of the
nodes when it is healthy (so you can see the versions of everything i'm
running):<br>
</div>
<div><br></div>
<blockquote class="gmail_quote">
# rabbitmqctl status<br>Status of node 'rabbit <at> ip-10-123-123-123' ...
<br>[{pid,28978},<br>&nbsp;{running_applications,<br>&nbsp; &nbsp; &nbsp;
[{rabbitmq_management,"RabbitMQ Management Console","3.1.0"},<br>&nbsp;
&nbsp; &nbsp; {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.1.0"},<br>
&nbsp; &nbsp; &nbsp; {webmachine,"webmachine","1.9.1-rmq3.1.0-
git52e62bc"},<br>&nbsp; &nbsp; &nbsp; {mochiweb,"MochiMedia Web
Server","2.3.1-rmq3.1.0-gitd541e9a"},<br>&nbsp; &nbsp; &nbsp;
{rabbitmq_management_agent,"RabbitMQ Management Agent","3.1.0"},<br>
&nbsp; &nbsp; &nbsp; {rabbit,"RabbitMQ","3.1.0"},<br>&nbsp; &nbsp; &nbsp;
{os_mon,"CPO &nbsp;CXC 138 46","2.2.11"},<br>&nbsp; &nbsp; &nbsp;
{inets,"INETS &nbsp;CXC 138 49","5.9.4"},<br>&nbsp; &nbsp; &nbsp;
{mnesia,"MNESIA &nbsp;CXC 138 12","4.8"},<br>
&nbsp; &nbsp; &nbsp; {amqp_client,"RabbitMQ AMQP Client","3.1.0"},
<br>&nbsp; &nbsp; &nbsp; {xmerl,"XML parser","1.3.3"},<br>&nbsp; &nbsp;
&nbsp; {sasl,"SASL &nbsp;CXC 138 11","2.3.1"},<br>&nbsp; &nbsp; &nbsp;
{stdlib,"ERTS &nbsp;CXC 138 10","1.19.1"},<br>
&nbsp; &nbsp; &nbsp; {kernel,"ERTS &nbsp;CXC 138 10","2.16.1"}]},
<br>&nbsp;{os,{unix,linux}},<br>&nbsp;{erlang_version,<br>&nbsp; &nbsp;
&nbsp;"Erlang R16B (erts-5.10.1) [source-05f1189] [64-bit] [smp:8:8] [async-
threads:30] [hipe] [kernel-poll:true]\n"},<br>
&nbsp;{memory,<br>&nbsp; &nbsp; &nbsp;[{total,60179048},<br>&nbsp; &nbsp;
&nbsp; {connection_procs,5274808},<br>&nbsp; &nbsp; &nbsp; {queue_procs,<a
href="tel:13107264" value="+13107264" target="_blank">13107264</a>},
<br>&nbsp; &nbsp; &nbsp; {plugins,354272},<br>&nbsp; &nbsp; &nbsp;
{other_proc,9741830},<br>
&nbsp; &nbsp; &nbsp; {mnesia,150456},<br>&nbsp; &nbsp; &nbsp;
{mgmt_db,12248},<br>&nbsp; &nbsp; &nbsp; {msg_index,34768},<br>&nbsp; &nbsp;
&nbsp; {other_ets,1148248},<br>&nbsp; &nbsp; &nbsp; {binary,4905584},
<br>&nbsp; &nbsp; &nbsp; {code,19565863},<br>&nbsp; &nbsp; &nbsp;
{atom,703377},<br>&nbsp; &nbsp; &nbsp; {other_system,5180330}]},<br>
&nbsp;{vm_memory_high_watermark,0.8},<br>&nbsp;
{vm_memory_limit,5828434329},<br>&nbsp;{disk_free_limit,1000000000},
<br>&nbsp;{disk_free,3953344512},<br>&nbsp;{file_descriptors,<br>&nbsp;
&nbsp; &nbsp;[{total_limit,924},{total_used,6},{sockets_limit,829},
{sockets_used,4}]},<br>
&nbsp;{processes,[{limit,1048576},{used,348}]},<br>&nbsp;{run_queue,0},
<br>&nbsp;{uptime,3796}]<br>...done.</blockquote>
<div><br></div>
<div>Note that I am using Erlang v16 from the Erlang Solutions repo (esl-
erlang package). &nbsp;I originally was using Erlang v14 from Ubuntu repos
(erlang-base), but decided to upgrade hoping it would resolve the issue
(which it did not). &nbsp;I have also tried running RabbitMQ 3.0.4 without
success. &nbsp;This issue seems to affect both versions.</div>
<div><br></div>
<div>Thanks,</div>
<div>Karl</div>
<br>_______________________________________________<br>
rabbitmq-discuss mailing list<br><a href="mailto:rabbitmq-
discuss at ...">rabbitmq-discuss <at> lists.rabbitmq.com</a><br><a
href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss"
target="_blank">https://lists.rabbitmq.com/cgi-
bin/mailman/listinfo/rabbitmq-discuss</a><br><br>
</blockquote>
</div>
<br>
</div>
</div>

Hi Brett,


That would explain why all hell breaks lose when the acks are being
subtracted, but I still don't understand what is causing the queue to
disappear in the first place. The only administrative actions performed on
the queues in all our code base are declares or passive declares (to check
for queue existence). We are just using Spring AMQP's
SimpleMessageListenerContainer (which, btw, the code for that class is
anything but "Simple") for consumers, and I've scanned through that code and
didn't find any place where it is dropping or destroying a queue or binding.


Is there are known bug where rabbit will cause a queue to disappear? All my
queues are durable and my messages are NOT persistent (we need high
throughput and loosing a few messages is not a big deal). I made the queue
durable after we first started noticing the crashes so the queue would come
back up on node restart. But durability doesn't seem to protect it from
disappearing.


Any thoughts on this would be greatly, greatly appreciated.


-Karl

Search Discussions

Discussion Posts

Previous

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 3 of 8 | next ›
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedMay 18, '13 at 3:20a
activeMay 20, '13 at 2:12p
posts8
users4
websiterabbitmq.com
irc#rabbitmq

People

Translate

site design / logo © 2017 Grokbase