[Questions] Quorum queue replica times out registering a consumer

[iamhritik]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I agree to provide all relevant information (versions, logs, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

3.13.7 or older

Erlang version used

26.2.x

Operating system (distribution) used

Ubuntu 22.04.5 LTS

How is RabbitMQ deployed?

Debian package

rabbitmq-diagnostics status output

Status of node rabbit@rmq-oa-replica-1 ...
Runtime

OS PID: 609
OS: Linux
Uptime (seconds): 45962
Is under maintenance?: false
RabbitMQ version: 3.13.3
RabbitMQ release series support status: supported
Node name: rabbit@rmq-oa-replica-1
Erlang configuration: Erlang/OTP 26 [erts-14.2.5.3] [source] [64-bit] [smp:16:16] [ds:16:16:10] [async-threads:1] [jit]
Crypto library: OpenSSL 3.0.2 15 Mar 2022
Erlang processes: 47826 used, 1048576 limit
Scheduler run queue: 3
Cluster heartbeat timeout (net_ticktime): 60

Plugins

Enabled plugin file: /etc/rabbitmq/enabled_plugins
Enabled plugins:

 * rabbitmq_prometheus
 * rabbitmq_shovel_management
 * rabbitmq_federation
 * rabbitmq_sharding
 * rabbitmq_peer_discovery_aws
 * rabbitmq_consistent_hash_exchange
 * accept
 * rabbitmq_peer_discovery_common
 * rabbitmq_random_exchange
 * rabbitmq_shovel
 * amqp10_client
 * prometheus
 * rabbitmq_aws
 * rabbitmq_web_mqtt
 * rabbitmq_mqtt
 * rabbitmq_management
 * rabbitmq_management_agent
 * rabbitmq_web_dispatch
 * amqp_client
 * cowboy
 * cowlib
 * oauth2_client
 * jose

Data directory

Node data directory: /var/lib/rabbitmq/mnesia/rabbit@rmq-oa-replica-1
Raft data directory: /var/lib/rabbitmq/mnesia/rabbit@rmq-oa-replica-1/quorum/rabbit@rmq-oa-replica-1

Config files

 * /etc/rabbitmq/rabbitmq.conf

Log file(s)

 * /var/log/rabbitmq/rabbit@rmq-oa-replica-1.log
 * <stdout>

Alarms

(none)

Memory

Total memory used: 4.5366 gb
Calculation strategy: rss
Memory high watermark setting: 0.9 of available memory, computed to: 119.3901 gb

binary: 17.6652 gb (67.31 %)
allocated_unused: 7.0024 gb (26.68 %)
mgmt_db: 0.3178 gb (1.21 %)
connection_other: 0.2516 gb (0.96 %)
quorum_ets: 0.2014 gb (0.77 %)
plugins: 0.1938 gb (0.74 %)
other_system: 0.1457 gb (0.56 %)
quorum_queue_procs: 0.1412 gb (0.54 %)
metrics: 0.0917 gb (0.35 %)
other_proc: 0.0821 gb (0.31 %)
other_ets: 0.0694 gb (0.26 %)
code: 0.0369 gb (0.14 %)
connection_channels: 0.0197 gb (0.08 %)
connection_writers: 0.0174 gb (0.07 %)
connection_readers: 0.0054 gb (0.02 %)
atom: 0.0019 gb (0.01 %)
msg_index: 0.0004 gb (0.0 %)
mnesia: 0.0003 gb (0.0 %)
metadata_store: 0.0002 gb (0.0 %)
metadata_store_ets: 0.0 gb (0.0 %)
quorum_queue_dlx_procs: 0.0 gb (0.0 %)
stream_queue_procs: 0.0 gb (0.0 %)
stream_queue_replica_reader_procs: 0.0 gb (0.0 %)
queue_procs: 0.0 gb (0.0 %)
queue_slave_procs: 0.0 gb (0.0 %)
stream_queue_coordinator_procs: 0.0 gb (0.0 %)
reserved_unallocated: 0.0 gb (0.0 %)

File Descriptors

Total: 45430, limit: 75439
Sockets: 45320, limit: 67893

Free Disk Space

Low free disk space watermark: 0.05 gb
Free disk space: 263.7334 gb

Totals

Connection count: 45093
Queue count: 43
Virtual host count: 1

Listeners

Interface: 0.0.0.0, port: 15672, protocol: http, purpose: HTTP API
Interface: 0.0.0.0, port: 1443, protocol: mqtt, purpose: MQTT
Interface: [::], port: 15675, protocol: http/web-mqtt, purpose: MQTT over WebSockets
Interface: [::], port: 15692, protocol: http/prometheus, purpose: Prometheus exporter API over HTTP
Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: 0.0.0.0, port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0

Logs from node 1 (with sensitive values edited out)

2024-12-06 20:15:57.540298+00:00 [error] <0.1722.0> Error on AMQP connection <0.1722.0> (192.168.1.0:51570 -> 192.30.110.250:5672, vhost: '/', user: 'admin', state: running), channel 1:
2024-12-06 20:15:57.540298+00:00 [error] <0.1722.0>  {{badmatch,{timeout,{'%2F_natco_cz_event','rabbit@rmq-oa-replica-5'}}},
2024-12-06 20:15:57.540298+00:00 [error] <0.1722.0>  [{rabbit_quorum_queue,consume,3,
2024-12-06 20:15:57.540298+00:00 [error] <0.1722.0>                        [{file,"rabbit_quorum_queue.erl"},{line,863}]},
2024-12-06 20:15:57.540298+00:00 [error] <0.1722.0>   {rabbit_queue_type,consume,3,[{file,"rabbit_queue_type.erl"},{line,430}]},
2024-12-06 20:15:57.540298+00:00 [error] <0.1722.0>   {rabbit_channel,'-basic_consume/8-fun-0-',10,
2024-12-06 20:15:57.540298+00:00 [error] <0.1722.0>                   [{file,"rabbit_channel.erl"},{line,1769}]},
2024-12-06 20:15:57.540298+00:00 [error] <0.1722.0>   {rabbit_misc,with_exit_handler,2,[{file,"rabbit_misc.erl"},{line,514}]},
2024-12-06 20:15:57.540298+00:00 [error] <0.1722.0>   {rabbit_channel,basic_consume,8,[{file,"rabbit_channel.erl"},{line,1766}]},
2024-12-06 20:15:57.540298+00:00 [error] <0.1722.0>   {rabbit_channel,handle_method,3,[{file,"rabbit_channel.erl"},{line,1429}]},
2024-12-06 20:15:57.540298+00:00 [error] <0.1722.0>   {rabbit_channel,handle_cast,2,[{file,"rabbit_channel.erl"},{line,649}]},
2024-12-06 20:15:57.540298+00:00 [error] <0.1722.0>   {gen_server2,handle_msg,2,[{file,"gen_server2.erl"},{line,1056}]}]}
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>     supervisor: {<0.4429.0>,rabbit_channel_sup}
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>     errorContext: child_terminated
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>     reason: {{badmatch,
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>                  {timeout,
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>                      {'%2F_natco_cz_event','rabbit@rmq-oa-replica-5'}}},
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>              [{rabbit_quorum_queue,consume,3,
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>                   [{file,"rabbit_quorum_queue.erl"},{line,863}]},
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>               {rabbit_queue_type,consume,3,
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>                   [{file,"rabbit_queue_type.erl"},{line,430}]},
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>               {rabbit_channel,'-basic_consume/8-fun-0-',10,
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>                   [{file,"rabbit_channel.erl"},{line,1769}]},
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>               {rabbit_misc,with_exit_handler,2,
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>                   [{file,"rabbit_misc.erl"},{line,514}]},
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>               {rabbit_channel,basic_consume,8,
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>                   [{file,"rabbit_channel.erl"},{line,1766}]},
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>               {rabbit_channel,handle_method,3,
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>                   [{file,"rabbit_channel.erl"},{line,1429}]},
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>               {rabbit_channel,handle_cast,2,
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>                   [{file,"rabbit_channel.erl"},{line,649}]},
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>               {gen_server2,handle_msg,2,
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>                   [{file,"gen_server2.erl"},{line,1056}]}]}
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>     offender: [{pid,<0.4432.0>},
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>                {id,channel},
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>                {mfargs,
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>                    {rabbit_channel,start_link,
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>                        [1,<0.1722.0>,<0.4430.0>,<0.1722.0>,
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>                         <<"192.168.1.0:51570 -> 192.30.110.250:5672">>,
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>                         rabbit_framing_amqp_0_9_1,
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>                         {user,<<"admin">>,
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>                             [administrator],
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>                             [{rabbit_auth_backend_internal,
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>                                  #Fun<rabbit_auth_backend_internal.3.26105038>}]},
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>                         <<"/">>,
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>                         [{<<"exchange_exchange_bindings">>,bool,true},
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>                          {<<"connection.blocked">>,bool,true},
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>                          {<<"authentication_failure_close">>,bool,true},
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>                          {<<"basic.nack">>,bool,true},
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>                          {<<"publisher_confirms">>,bool,true},
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>                          {<<"consumer_cancel_notify">>,bool,true}],
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>                         <0.1723.0>,<0.4431.0>]}},
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>                {restart_type,transient},
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>                {significant,true},
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>                {shutdown,70000},
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>                {child_type,worker}]
2024-12-06 20:15:57.540201+00:00 [error] <0.4429.0>
2024-12-06 20:15:57.540424+00:00 [warning] <0.1722.0> Non-AMQP exit reason '{{badmatch,
2024-12-06 20:15:57.540424+00:00 [warning] <0.1722.0>                            {timeout,
2024-12-06 20:15:57.540424+00:00 [warning] <0.1722.0>                                {'%2F_natco_cz_event',
2024-12-06 20:15:57.540424+00:00 [warning] <0.1722.0>                                    'rabbit@rmq-oa-replica-5'}}},
2024-12-06 20:15:57.540424+00:00 [warning] <0.1722.0>                        [{rabbit_quorum_queue,consume,3,
2024-12-06 20:15:57.540424+00:00 [warning] <0.1722.0>                             [{file,"rabbit_quorum_queue.erl"},{line,863}]},
2024-12-06 20:15:57.540424+00:00 [warning] <0.1722.0>                         {rabbit_queue_type,consume,3,
2024-12-06 20:15:57.540424+00:00 [warning] <0.1722.0>                             [{file,"rabbit_queue_type.erl"},{line,430}]},
2024-12-06 20:15:57.540424+00:00 [warning] <0.1722.0>                         {rabbit_channel,'-basic_consume/8-fun-0-',10,
2024-12-06 20:15:57.540424+00:00 [warning] <0.1722.0>                             [{file,"rabbit_channel.erl"},{line,1769}]},
2024-12-06 20:15:57.540424+00:00 [warning] <0.1722.0>                         {rabbit_misc,with_exit_handler,2,
2024-12-06 20:15:57.540424+00:00 [warning] <0.1722.0>                             [{file,"rabbit_misc.erl"},{line,514}]},
2024-12-06 20:15:57.540424+00:00 [warning] <0.1722.0>                         {rabbit_channel,basic_consume,8,
2024-12-06 20:15:57.540424+00:00 [warning] <0.1722.0>                             [{file,"rabbit_channel.erl"},{line,1766}]},
2024-12-06 20:15:57.540424+00:00 [warning] <0.1722.0>                         {rabbit_channel,handle_method,3,
2024-12-06 20:15:57.540424+00:00 [warning] <0.1722.0>                             [{file,"rabbit_channel.erl"},{line,1429}]},
2024-12-06 20:15:57.540424+00:00 [warning] <0.1722.0>                         {rabbit_channel,handle_cast,2,
2024-12-06 20:15:57.540424+00:00 [warning] <0.1722.0>                             [{file,"rabbit_channel.erl"},{line,649}]},
2024-12-06 20:15:57.540424+00:00 [warning] <0.1722.0>                         {gen_server2,handle_msg,2,
2024-12-06 20:15:57.540424+00:00 [warning] <0.1722.0>                             [{file,"gen_server2.erl"},{line,1056}]}]}'
2024-12-06 20:15:57.540564+00:00 [error] <0.4429.0>     supervisor: {<0.4429.0>,rabbit_channel_sup}
2024-12-06 20:15:57.540564+00:00 [error] <0.4429.0>     errorContext: shutdown
2024-12-06 20:15:57.540564+00:00 [error] <0.4429.0>     reason: reached_max_restart_intensity
2024-12-06 20:15:57.540564+00:00 [error] <0.4429.0>     offender: [{pid,<0.4432.0>},
2024-12-06 20:15:57.540564+00:00 [error] <0.4429.0>                {id,channel},
2024-12-06 20:15:57.540564+00:00 [error] <0.4429.0>                {mfargs,
2024-12-06 20:15:57.540564+00:00 [error] <0.4429.0>                    {rabbit_channel,start_link,
2024-12-06 20:15:57.540564+00:00 [error] <0.4429.0>                        [1,<0.1722.0>,<0.4430.0>,<0.1722.0>,
2024-12-06 20:15:57.540564+00:00 [error] <0.4429.0>                         <<"192.168.1.0:51570 -> 192.30.110.250:5672">>,
2024-12-06 20:15:57.540564+00:00 [error] <0.4429.0>                         rabbit_framing_amqp_0_9_1,
2024-12-06 20:15:57.540564+00:00 [error] <0.4429.0>                         {user,<<"admin">>,
2024-12-06 20:15:57.540564+00:00 [error] <0.4429.0>                             [administrator],
2024-12-06 20:15:57.540564+00:00 [error] <0.4429.0>                             [{rabbit_auth_backend_internal,
2024-12-06 20:15:57.540564+00:00 [error] <0.4429.0>                                  #Fun<rabbit_auth_backend_internal.3.26105038>}]},
2024-12-06 20:15:57.540564+00:00 [error] <0.4429.0>                         <<"/">>,
2024-12-06 20:15:57.540564+00:00 [error] <0.4429.0>                         [{<<"exchange_exchange_bindings">>,bool,true},
2024-12-06 20:15:57.540564+00:00 [error] <0.4429.0>                          {<<"connection.blocked">>,bool,true},
2024-12-06 20:15:57.540564+00:00 [error] <0.4429.0>                          {<<"authentication_failure_close">>,bool,true},
2024-12-06 20:15:57.540564+00:00 [error] <0.4429.0>                          {<<"basic.nack">>,bool,true},
2024-12-06 20:15:57.540564+00:00 [error] <0.4429.0>                          {<<"publisher_confirms">>,bool,true},
2024-12-06 20:15:57.540564+00:00 [error] <0.4429.0>                          {<<"consumer_cancel_notify">>,bool,true}],
2024-12-06 20:15:57.540564+00:00 [error] <0.4429.0>                         <0.1723.0>,<0.4431.0>]}},
2024-12-06 20:15:57.540564+00:00 [error] <0.4429.0>                {restart_type,transient},
2024-12-06 20:15:57.540564+00:00 [error] <0.4429.0>                {significant,true},
2024-12-06 20:15:57.540564+00:00 [error] <0.4429.0>                {shutdown,70000},
2024-12-06 20:15:57.540564+00:00 [error] <0.4429.0>                {child_type,worker}]
2024-12-06 20:15:57.540564+00:00 [error] <0.4429.0>
2024-12-06 20:15:57.542284+00:00 [info] <0.1722.0> closing AMQP connection <0.1722.0> (192.168.1.0:51570 -> 192.30.110.250:5672, vhost: '/', user: 'admin')
2024-12-06 20:15:57.561833+00:00 [info] <0.7224.0> accepting AMQP connection <0.7224.0> (192.168.1.0:46356 -> 192.30.110.250:5672)

Logs from node 2 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 3 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

rabbitmq.conf

listeners.tcp.1 = 0.0.0.0:5672
mqtt.listeners.tcp.1 = 0.0.0.0:1443
mqtt.tcp_listen_options.backlog = 9182
mqtt.tcp_listen_options.recbuf  = 191072
mqtt.tcp_listen_options.sndbuf  = 191072
mqtt.tcp_listen_options.keepalive = true
mqtt.tcp_listen_options.nodelay   = true
mqtt.tcp_listen_options.exit_on_close = true
mqtt.tcp_listen_options.send_timeout  = 120
# anonymous connections, if allowed, will use the default
# credentials specified here
mqtt.allow_anonymous  = false
mqtt.default_user     = guest
mqtt.default_pass     = guest
mqtt.vhost            = /
mqtt.exchange         = amq.topic
# 24 hours by default
mqtt.max_session_expiry_interval_seconds = 86400
#mqtt.prefetch         = 10
management.listener.port = 15672
management.listener.ip   = 0.0.0.0

###### Disk & Memory Configuration ######
disk_free_limit.absolute = 50MB
vm_memory_high_watermark.relative = 0.9
######## Logging #######
log.file.level = info
handshake_timeout = 20000
tcp_listen_options.backlog = 9182
tcp_listen_options.recbuf  = 191072
tcp_listen_options.sndbuf  = 191072
tcp_listen_options.keepalive = true
tcp_listen_options.nodelay   = true

Steps to deploy RabbitMQ cluster

RabbitMQ Official Doc steps to install RabbitMQ on Ubuntu machines

Steps to reproduce the behavior in question

When the NiFi consumers were stopped due to [XYZ reason] and all the messages were in a ready state, once the consumers resumed and started consuming a large number of messages (around 3.5 million), I started seeing errors in the logs and was unable to consume any messages from the RMQ nodes

advanced.config

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location

# PASTE advanced.config HERE, BETWEEN BACKTICKS

Application code

# PASTE CODE HERE, BETWEEN BACKTICKS

Kubernetes deployment file

# Relevant parts of K8S deployment that demonstrate how RabbitMQ is deployed
# PASTE YAML HERE, BETWEEN BACKTICKS

What problem are you trying to solve?

To able to consume messages that are in Ready State on RabbitMQ Cluster

[michaelklishin]

The exception means that a quorum queue replica times out registering a consumer. There are no details as to why that might be, the queue can have an online minority at that time.

Unless there is a way to reproduce against 4.0.4, that's all we can suggest because 3.13.x is out of community support.

[michaelklishin]

rabbitmq-queues quorum_status [--vhost <name>] [queue name]

output and logs from all nodes would be necessary if there is a known way to reproduce this at least some of the time.