Quorum queue behavior when a cluster member is replaced (a node is deleted and another one is added with the same name)

[christophermlewis]

Describe the bug

RabbitMQ 3.13.7Erlang 26.0

We have a working 3 node cluster that we wish to migrate to a new AWS VPC.
To do this we wish to add new nodes to the existing cluster in a new VPC and enable VPC peering.

With the newly created rabbit servers in the new AWS VPC and enabled vpc peering to the existing working cluster.
The additional 3 nodes are able to join the existing cluster, all metrics indicate the network is good.
We are able to re-balance the quorum queues and each node across the two separate VPC's are able to communicate. We can see this from the successful clustering and the individual logs showing communication across the cluster.

Adding the new nodes to the cluster did not cause any problems, amqp connections were able to be established.
However As soon as we put one of the old nodes into maintenance mode and a client tries to consume a queue we we get ** (EXIT) shutdown: {:connection_closing, {:server_initiated_close, 541, "INTERNAL_ERROR"}} and in the rabbit log a tuple of death.
We have tried rabbitmq-queues shrink, rabbitmq-queues grow but admit our knowledge of how quroum queues should work and be managed is unclear to us. We have read the documentation and from what we can tell, what I have described shouldn't cause the cluster to start failing. The cluster isn't complete failed, it just seems certain connections needing certain queues causes the problem. Speculating it is as though the new node isn't able to find the queue the client is looking for even though we have 2 of the old nodes that are members of the quorum.

{"time":"2025-01-07 08:44:24.139987+00:00","level":"error","msg":"** Generic server <0.7162637.0> terminating\n** Last message in was {'$gen_cast',\n {method,\n {'basic.consume',0,<<\"SERVICE.TOPOSERVICE\">>,\n <<>>,false,false,false,false,[]},\n none,noflow}}\n** When Server state == {ch,\n {conf,running,rabbit_framing_amqp_0_9_1,11,\n <0.7162447.0>,<0.7162639.0>,<0.7162447.0>,\n <<\"10.0.10.107:52830 -> 10.1.1.193:5671\">>,\n undefined,\n {user,<<\"dev-mdg-services-dmz\">>,[],\n [{rabbit_auth_backend_internal,\n #Fun<rabbit_auth_backend_internal.3.26105038>}]},\n <<\"dev-mdg-dmz\">>,<<\"OUTGOING.SERVICE.QUEUE\">>,\n <0.7162414.0>,\n [{<<\"publisher_confirms\">>,bool,true},\n {<<\"exchange_exchange_bindings\">>,bool,true},\n {<<\"basic.nack\">>,bool,true},\n {<<\"consumer_cancel_notify\">>,bool,true},\n {<<\"connection.blocked\">>,bool,true},\n {<<\"authentication_failure_close\">>,bool,true}],\n none,20,134217728,1800000,#{},infinity,1000000000,\n false},\n {lstate,<0.7162640.0>,false},\n none,1,\n {0,[],[]},\n {state,#{},erlang},\n #{},#{},\n {state,fine,5000,#Ref<0.4254837829.1655963650.33629>},\n false,1,\n {rabbit_confirms,undefined,#{}},\n [],[],none,flow,[],\n {rabbit_queue_type,#{}},\n #Ref<0.4254837829.1655963650.33624>,false}\n** Reason for termination == \n** {{badmatch,{error,noproc}},\n [{rabbit_quorum_queue,consume,3,\n [{file,\"rabbit_quorum_queue.erl\"},{line,875}]},\n {rabbit_queue_type,consume,3,[{file,\"rabbit_queue_type.erl\"},{line,479}]},\n {rabbit_channel,'-basic_consume/8-fun-0-',10,\n [{file,\"rabbit_channel.erl\"},{line,1778}]},\n {rabbit_misc,with_exit_handler,2,[{file,\"rabbit_misc.erl\"},{line,515}]},\n {rabbit_channel,basic_consume,8,\n [{file,\"rabbit_channel.erl\"},{line,1775}]},\n {rabbit_channel,handle_method,3,\n [{file,\"rabbit_channel.erl\"},{line,1438}]},\n {rabbit_channel,handle_cast,2,[{file,\"rabbit_channel.erl\"},{line,659}]},\n {gen_server2,handle_msg,2,[{file,\"gen_server2.erl\"},{line,1056}]}]}\n","pid":"<0.7162637.0>"} {"time":"2025-01-07 08:44:24.140568+00:00","level":"error","msg":" crasher:\n initial call: rabbit_channel:init/1\n pid: <0.7162637.0>\n registered_name: []\n exception exit: {{badmatch,{error,noproc}},\n [{rabbit_quorum_queue,consume,3,\n [{file,\"rabbit_quorum_queue.erl\"},{line,875}]},\n {rabbit_queue_type,consume,3,\n [{file,\"rabbit_queue_type.erl\"},{line,479}]},\n {rabbit_channel,'-basic_consume/8-fun-0-',10,\n [{file,\"rabbit_channel.erl\"},{line,1778}]},\n {rabbit_misc,with_exit_handler,2,\n [{file,\"rabbit_misc.erl\"},{line,515}]},\n {rabbit_channel,basic_consume,8,\n [{file,\"rabbit_channel.erl\"},{line,1775}]},\n {rabbit_channel,handle_method,3,\n [{file,\"rabbit_channel.erl\"},{line,1438}]},\n {rabbit_channel,handle_cast,2,\n [{file,\"rabbit_channel.erl\"},{line,659}]},\n {gen_server2,handle_msg,2,\n [{file,\"gen_server2.erl\"},{line,1056}]}]}\n in function gen_server2:terminate/3 (gen_server2.erl, line 1172)\n ancestors: [<0.7162638.0>,<0.7162464.0>,<0.7162448.0>,<0.7162445.0>,\n <0.1219888.0>,<0.1219887.0>,<0.1219886.0>,<0.1219884.0>,\n <0.1219883.0>,rabbit_sup,<0.1209770.0>]\n message_queue_len: 4\n messages: [{'$gen_cast',\n {command,\n {'basic.consume_ok',\n <<\"amq.ctag-52RbEBUks0L0RM6Ifu4kpA\">>}}},\n emit_stats,emit_gen_server2_stats,tick]\n links: [<0.7162638.0>]\n dictionary: [{rand_seed,\n {#{max => 288230376151711743,type => exsplus,\n next => #Fun<rand.5.65977474>,\n jump => #Fun<rand.3.65977474>},\n [2407078749 5705299|119882446621519867]}},\n {topic_permission_cache,\n [{{resource,<<\"dev-mdg-dmz\">>,topic,<<\"DMZ\">>},\n #{routing_key => <<\"SERVICE.TOPOSERVICE\">>,\n variable_map =>\n #{<<\"username\">> => <<\"dev-mdg-services-dmz\">>,\n <<\"vhost\">> => <<\"dev-mdg-dmz\">>}},\n read}]},\n {{xtype_to_module,direct},rabbit_exchange_type_direct},\n {process_name,\n {rabbit_channel,\n {<<\"10.0.10.107:52830 -> 10.1.1.193:5671\">>,11}}},\n {permission_cache,\n [{{resource,<<\"dev-mdg-dmz\">>,queue,\n <<\"SERVICE.TOPOSERVICE\">>},\n #{},read},\n {{resource,<<\"dev-mdg-dmz\">>,exchange,<<\"DMZ\">>},\n #{},read},\n {{resource,<<\"dev-mdg-dmz\">>,queue,\n <<\"SERVICE.TOPOSERVICE\">>},\n #{},write},\n {{resource,<<\"dev-mdg-dmz\">>,exchange,\n <<\"OUTGOING.SERVICE\">>},\n #{},read},\n {{resource,<<\"dev-mdg-dmz\">>,queue,\n <<\"OUTGOING.SERVICE.QUEUE\">>},\n #{},write},\n {{resource,<<\"dev-m dg-dmz\">>,queue,\n <<\"OUTGOING.SERVICE.QUEUE\">>},\n #{},configure},\n {{resource,<<\"dev-mdg-dmz\">>,exchange,<<\"PARTNER\">>},\n #{},configure},\n {{resource,<<\"dev-mdg-dmz\">>,exchange,\n <<\"OUTGOING.SERVICE\">>},\n #{},configure},\n {{resource,<<\"dev-mdg-dmz\">>,queue,\n <<\"SERVICE.TOPOSERVICE\">>},\n #{},configure},\n {{resource,<<\"dev-mdg-dmz\">>,exchange,<<\"DMZ\">>},\n #{},configure}]},\n {{xtype_to_module,topic},rabbit_exchange_type_to pic},\n {guid_secure,\n {{0,'rabbit@ip-10-1-1-193',\n #Ref<0.4254837829.1655963650.33735>},\n 0}},\n {channel_operation_timeout,15000},\n {permission_cache_can_expire,false}]\n trap_exit: true\n status: running\n heap_size: 6772\n stack_size: 28\n reductions: 28899\n neighbours:\n","line":584,"pid":"<0.7162637.0>","file":"proc_lib.erl","domain":"otp.sasl","mfa":["proc_lib","crash_report",4]}

Reproduction steps

We will create the setup described in a sandbox environment and update this part if we can reproduce it outside the affected environment

Expected behavior

We don't expect the consumer to crash when consuming a quorum queue that has 2 members available from a node that isn't a member.

Additional context

No response

[christophermlewis]

Ok further testing shows that the problem is not the peering.
With the original 3 node cluster, if we remove one of the nodes, reset it and add it back to the cluster, if a consumer connects to it and tries to consume a quorum queue then we get the same tuple of death

[michaelklishin]

RabbitMQ 3.13.x is out of community support.

if we remove one of the nodes, reset it and add it back to the cluster, if a consumer connects to it > and tries to consume a quorum queue then we get the same tuple of death

Quorum queues are highly sensitive to node identity changes. If a node is about to be permanently removed from the cluster and replaced, it must be removed from the list of replicas of all quorum queues with rabbitmq-queues shrink and such.

Then, after a new node is added, QQ replicas can be added to it using rabbitmq-queues grow.

Here is a highly relevant documentation section.