RCA for an exception on a RabbitMQ 3.12 node #12781

rayate2410 · 2024-11-22T07:42:11Z

rayate2410
Nov 22, 2024

Community Support Policy

I have read RabbitMQ's Community Support Policy
I agree to provide all relevant information (versions, logs, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

3.13.7 or older

Erlang version used

26.0.x

Operating system (distribution) used

linux

How is RabbitMQ deployed?

Community Docker image

rabbitmq-diagnostics status output

See https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics

RabbitMQ version: RabbitMQ 3.12.12

application.rabbitmq-ha.log: [1730293923.144110619, {"time":"2024-10-30 14:12:03.143552+01:00","level":"notice","msg":"SIGTERM received - shutting down\n","pid":"<0.61.0>"}]
application.rabbitmq-ha.log: [1730293923.250299156, {"time":"2024-10-30 14:12:03.246970+01:00","level":"error","msg":"** Generic server <0.1263.0> terminating \n** Last message in was {'EXIT',<0.1259.0>,shutdown}\n** When Server state == {state,amqp_direct_connection,\n                         {state,\n                          'rabbit@eisrabbitmq-rabbitmq-ha-0.eisrabbitmq-rabbitmq-ha-discovery.edison-system.svc.cluster.local',\n                          {user,<<\"LimitedUser\">>,\n                           [administrator],\n                           [{rabbit_auth_backend_internal,\n                             #Fun<rabbit_auth_backend_internal.3.20918515>}]},\n                          <<\"ServiceTools\">>,\n                          {amqp_params_direct,<<\"LimitedUser\">>,\n                           {encrypted,\n                            <<\"/l31Id8GJUXjQ2/zWQm0R6jxn6hA3FYffz9qwyAAVUVzQKL/pUjOYCIbgCKfsuT9aaySlBEUkZmZwHZQdL8mqNhh9KvssQhS1I3tyaF0W0Y=\">>},\n                           <<\"ServiceTools\">>,\n                           'rabbit@eisrabbitmq-rabbitmq-ha-0.eisrabbitmq-rabbitmq-ha-discovery.edison-system.svc.cluster.local',\n                           {amqp_adapter_info,\n                            {198,18,95,195},\n                            15674,\n                            {198,18,95,202},\n                            55256,\n                            <<\"198.18.95.202:55256 -> 198.18.95.195:15674\">>,\n                            {'Web STOMP',\"1.2\"},\n                            [{channels,1},\n                             {channel_max,1},\n                             {frame_max,0},\n                             {client_properties,\n                              [{<<\"product\">>,longstr,<<\"STOMP client\">>}]},\n                             {ssl,false}]},\n                           []},\n                          {amqp_adapter_info,\n                           {198,18,95,195},\n                           15674,\n                           {198,18,95,202},\n                           55256,\n                           <<\"198.18.95.202:55256 -> 198.18.95.195:15674\">>,\n                           {'Web STOMP',\"1.2\"},\n                           [{channels,1},\n                            {channel_max,1},\n                            {frame_max,0},\n                            {client_properties,\n                             [{<<\"product\">>,longstr,<<\"STOMP client\">>}]},\n                            {ssl,false}]},\n                          <0.1268.0>,undefined,1730293650929},\n                         <0.1267.0>,\n                         {amqp_params_direct,<<\"LimitedUser\">>,\n                          {encrypted,\n                           <<\"/l31Id8GJUXjQ2/zWQm0R6jxn6hA3FYffz9qwyAAVUVzQKL/pUjOYCIbgCKfsuT9aaySlBEUkZmZwHZQdL8mqNhh9KvssQhS1I3tyaF0W0Y=\">>},\n                          <<\"ServiceTools\">>,\n                          'rabbit@eisrabbitmq-rabbitmq-ha-0.eisrabbitmq-rabbitmq-ha-discovery.edison-system.svc.cluster.local',\n                          {amqp_adapter_info,\n                           {198,18,95,195},\n                           15674,\n                           {198,18,95,202},\n                           55256,\n                           <<\"198.18.95.202:55256 -> 198.18.95.195:15674\">>,\n                           {'Web STOMP',\"1.2\"},\n                           [{channels,1},\n                            {channel_max,1},\n                            {frame_max,0},\n                            {client_properties,\n                             [{<<\"product\">>,longstr,<<\"STOMP client\">>}]},\n                            {ssl,false}]},\n                          []},\n                         0,\n                         [{<<\"capabilities\">>,table,\n                           [{<<\"publisher_confirms\">>,bool,true},\n                            {<<\"exchange_exchange_bindings\">>,bool,true},\n                            {<<\"basic.nack\">>,bool,true},\n                            {<<\"consumer_cancel_notify\">>,bool,true},\n                            {<<\"connection.blocked\">>,bool,true},\n                            {<<\"consumer_priorities\">>,bool,true},\n                            {<<\"authentication_failure_close\">>,bool,true},\n                            {<<\"per_consumer_qos\">>,bool,true},\n                            {<<\"direct_reply_to\">>,bool,true}]},\n                          {<<\"cluster_name\">>,longstr,\n                           <<\"rabbit@eisrabbitmq-rabbitmq-ha-0.eisrabbitmq-rabbitmq-ha-discovery.edison-system.svc.cluster.local\">>},\n                          {<<\"copyright\">>,longstr,\n                           <<\"Copyright (c) 2007-2023 Broadcom Inc and/or its subsidiaries\">>},\n                          {<<\"information\">>,longstr,\n                           <<\"Licensed under the MPL 2.0. Website: https://rabbitmq.com\">>},\n                          {<<\"platform\">>,longstr,<<\"Erlang/OTP 25.3.2.8\">>},\n                          {<<\"product\">>,longstr,<<\"RabbitMQ\">>},\n                          {<<\"version\">>,longstr,<<\"3.12.12\">>}],\n                         none,false}\n** Reason for termination ==\n** \"stopping because dependent process <0.1259.0> died: shutdown\"\n","domain":"otp","file":"gen_server.erl","line":1402,"mfa":["gen_server","error_info",8],"pid":"<0.1263.0>"}]
application.rabbitmq-ha.log: [1730293923.250365079, {"time":"2024-10-30 14:12:03.249684+01:00","level":"error","msg":"  crasher:\n    initial call: amqp_gen_connection:init/1\n    pid: <0.1263.0>\n    registered_name: []\n    exception exit: \"stopping because dependent process <0.1259.0> died: shutdown\"\n      in function  gen_server:handle_common_reply/8 (gen_server.erl, line 1241)\n    ancestors: [<0.1261.0>,amqp_sup,<0.921.0>]\n    message_queue_len: 0\n    messages: []\n    links: [<0.1261.0>]\n    dictionary: [{process_name,\n                      {amqp_gen_connection,\n                          <<\"198.18.95.202:55256 -> 198.18.95.195:15674\">>}},\n                  {gen_server_call_timeout,130000}]\n    trap_exit: true\n    status: running\n    heap_size: 4185\n    stack_size: 28\n    reductions: 51983\n  neighbours:\n","domain":"otp.sasl","file":"proc_lib.erl","line":539,"mfa":["proc_lib","crash_report",4],"pid":"<0.1263.0>"}]
application.rabbitmq-ha.log: [1730293923.251287305, {"time":"2024-10-30 14:12:03.250356+01:00","level":"info","msg":"stopped STOMP TCP listener on [::]:61613","pid":"<0.1027.0>"}]
application.rabbitmq-ha.log: [1730293923.251334655, {"time":"2024-10-30 14:12:03.250297+01:00","level":"error","msg":"    supervisor: {<0.1261.0>,amqp_connection_sup}\n    errorContext: child_terminated\n    reason: \"stopping because dependent process <0.1259.0> died: shutdown\"\n    offender: [{pid,<0.1263.0>},\n               {id,connection},\n               {mfargs,\n                   {amqp_gen_connection,start_link,\n                       [<0.1262.0>,\n                        {amqp_params_direct,<<\"LimitedUser\">>,\n                            {encrypted,\n                                <<\"/l31Id8GJUXjQ2/zWQm0R6jxn6hA3FYffz9qwyAAVUVzQKL/pUjOYCIbgCKfsuT9aaySlBEUkZmZwHZQdL8mqNhh9KvssQhS1I3tyaF0W0Y=\">>},\n                            <<\"ServiceTools\">>,\n                            'rabbit@eisrabbitmq-rabbitmq-ha-0.eisrabbitmq-rabbitmq-ha-discovery.edison-system.svc.cluster.local',\n                            {amqp_adapter_info,\n                                {198,18,95,195},\n                                15674,\n                                {198,18,95,202},\n                                55256,\n                                <<\"198.18.95.202:55256 -> 198.18.95.195:15674\">>,\n                                {'Web STOMP',\"1.2\"},\n                                [{channels,1},\n                                 {channel_max,1},\n                                 {frame_max,0},\n                                 {client_properties,\n                                     [{<<\"product\">>,longstr,\n                                       <<\"STOMP client\">>}]},\n                                 {ssl,false}]},\n                            []}]}},\n               {restart_type,transient},\n               {significant,true},\n               {shutdown,brutal_kill},\n               {child_type,worker}]\n","domain":"otp.sasl","file":"supervisor.erl","line":754,"mfa":["supervisor","do_restart",3],"pid":"<0.1261.0>"}]
application.rabbitmq-ha.log: [1730293923.343394649, {"time":"2024-10-30 14:12:03.251047+01:00","level":"error","msg":"    supervisor: {<0.1261.0>,amqp_connection_sup}\n    errorContext: shutdown\n    reason: reached_max_restart_intensity\n    offender: [{pid,<0.1263.0>},\n               {id,connection},\n               {mfargs,\n                   {amqp_gen_connection,start_link,\n                       [<0.1262.0>,\n                        {amqp_params_direct,<<\"LimitedUser\">>,\n                            {encrypted,\n                                <<\"/l31Id8GJUXjQ2/zWQm0R6jxn6hA3FYffz9qwyAAVUVzQKL/pUjOYCIbgCKfsuT9aaySlBEUkZmZwHZQdL8mqNhh9KvssQhS1I3tyaF0W0Y=\">>},\n                            <<\"ServiceTools\">>,\n                            'rabbit@eisrabbitmq-rabbitmq-ha-0.eisrabbitmq-rabbitmq-ha-discovery.edison-system.svc.cluster.local',\n                            {amqp_adapter_info,\n                                {198,18,95,195},\n                                15674,\n                                {198,18,95,202},\n                                55256,\n                                <<\"198.18.95.202:55256 -> 198.18.95.195:15674\">>,\n                                {'Web STOMP',\"1.2\"},\n                                [{channels,1},\n                                 {channel_max,1},\n                                 {frame_max,0},\n                                 {client_properties,\n                                     [{<<\"product\">>,longstr,\n                                       <<\"STOMP client\">>}]},\n                                 {ssl,false}]},\n                            []}]}},\n               {restart_type,transient},\n               {significant,true},\n               {shutdown,brutal_kill},\n               {child_type,worker}]\n","domain":"otp.sasl","file":"supervisor.erl","line":811,"mfa":["supervisor","restart",2],"pid":"<0.1261.0>"}]
application.rabbitmq-ha.log: [1730293923.548933825, {"time":"2024-10-30 14:12:03.460517+01:00","level":"warning","msg":"HTTP listener registry could not find context rabbitmq_management_tls","domain":"rabbitmq","pid":"<0.927.0>"}]
application.rabbitmq-ha.log: [1730293923.562371060, {"time":"2024-10-30 14:12:03.561907+01:00","level":"info","msg":"Will unregister with peer discovery backend rabbit_peer_discovery_k8s","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293923.563057951, {"time":"2024-10-30 14:12:03.562235+01:00","level":"info","msg":"stopped TLS (SSL) listener on [::]:5671","pid":"<0.1113.0>"}]
application.rabbitmq-ha.log: [1730293923.648043831, {"time":"2024-10-30 14:12:03.647530+01:00","level":"info","msg":"stopped TCP listener on [::]:5672","pid":"<0.1093.0>"}]
application.rabbitmq-ha.log: [1730293923.650056156, {"time":"2024-10-30 14:12:03.649317+01:00","level":"error","msg":"Error on AMQP connection <0.1306.0> (198.18.170.2:61574 -> 198.18.95.195:5672, vhost: 'ServiceTools', user: 'ApplistUser', state: running), channel 0:\n operation none caused a connection exception connection_forced: \"broker forced connection closure with reason 'shutdown'\"","domain":"rabbitmq.connection","pid":"<0.1306.0>"}]

Logs from node 1 (with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

application.rabbitmq-ha.log: [1730293609.068252894, {"stream":"stdout","logtag":"F","log":"2024-10-30 14:06:48.971003+01:00 [warning] <0.132.0> Overriding Erlang cookie using the value set in the environment"}]
application.rabbitmq-ha.log: [1730293615.231193654, {"stream":"stdout","logtag":"F","log":"2024-10-30 14:06:55.230870+01:00 [notice] <0.44.0> Application syslog exited with reason: stopped"}]
application.rabbitmq-ha.log: [1730293615.231238255, {"stream":"stdout","logtag":"F","log":"2024-10-30 14:06:55.231030+01:00 [notice] <0.230.0> Logging: switching to configured handler(s); following messages may not be visible in this log output"}]
application.rabbitmq-ha.log: [1730293615.252340181, {"time":"2024-10-30 14:06:55.232068+01:00","level":"notice","msg":"Logging: configured log handlers are now ACTIVE","domain":"rabbitmq.prelaunch","file":"rabbit_prelaunch_logging.erl","line":1549,"mfa":["rabbit_prelaunch_logging","install_handlers",0],"pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293616.344047855, {"time":"2024-10-30 14:06:56.343368+01:00","level":"info","msg":"ra: starting system quorum_queues","domain":"ra","file":"src/ra_system.erl","line":67,"mfa":["ra_system","start",1],"pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293616.344111582, {"time":"2024-10-30 14:06:56.343600+01:00","level":"info","msg":"starting Ra system: quorum_queues in directory: /var/lib/rabbitmq/mnesia/rabbit@eisrabbitmq-rabbitmq-ha-0.eisrabbitmq-rabbitmq-ha-discovery.edison-system.svc.cluster.local/quorum/rabbit@eisrabbitmq-rabbitmq-ha-0.eisrabbitmq-rabbitmq-ha-discovery.edison-system.svc.cluster.local","domain":"ra","file":"src/ra_systems_sup.erl","line":31,"mfa":["ra_systems_sup","start_system",1],"pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293616.448812005, {"time":"2024-10-30 14:06:56.448336+01:00","level":"info","msg":"ra system 'quorum_queues' running pre init for 0 registered servers","domain":"ra","file":"src/ra_log_pre_init.erl","line":42,"mfa":["ra_log_pre_init","init",1],"pid":"<0.335.0>"}]
application.rabbitmq-ha.log: [1730293616.473259662, {"time":"2024-10-30 14:06:56.472831+01:00","level":"info","msg":"ra: meta data store initialised for system quorum_queues. 0 record(s) recovered","domain":"ra","file":"src/ra_log_meta.erl","line":58,"mfa":["ra_log_meta","init",1],"pid":"<0.336.0>"}]
application.rabbitmq-ha.log: [1730293616.500656442, {"time":"2024-10-30 14:06:56.500207+01:00","level":"notice","msg":"WAL: ra_log_wal init, open tbls: ra_log_open_mem_tables, closed tbls: ra_log_closed_mem_tables","domain":"ra","file":"src/ra_log_wal.erl","line":250,"mfa":["ra_log_wal","init",1],"pid":"<0.341.0>"}]
application.rabbitmq-ha.log: [1730293616.551940692, {"time":"2024-10-30 14:06:56.551621+01:00","level":"info","msg":"ra: starting system coordination","domain":"ra","file":"src/ra_system.erl","line":67,"mfa":["ra_system","start",1],"pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293616.552059683, {"time":"2024-10-30 14:06:56.551774+01:00","level":"info","msg":"starting Ra system: coordination in directory: /var/lib/rabbitmq/mnesia/rabbit@eisrabbitmq-rabbitmq-ha-0.eisrabbitmq-rabbitmq-ha-discovery.edison-system.svc.cluster.local/coordination/rabbit@eisrabbitmq-rabbitmq-ha-0.eisrabbitmq-rabbitmq-ha-discovery.edison-system.svc.cluster.local","domain":"ra","file":"src/ra_systems_sup.erl","line":31,"mfa":["ra_systems_sup","start_system",1],"pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293616.567353966, {"time":"2024-10-30 14:06:56.567020+01:00","level":"info","msg":"ra system 'coordination' running pre init for 0 registered servers","domain":"ra","file":"src/ra_log_pre_init.erl","line":42,"mfa":["ra_log_pre_init","init",1],"pid":"<0.348.0>"}]
application.rabbitmq-ha.log: [1730293616.568519996, {"time":"2024-10-30 14:06:56.568256+01:00","level":"info","msg":"ra: meta data store initialised for system coordination. 0 record(s) recovered","domain":"ra","file":"src/ra_log_meta.erl","line":58,"mfa":["ra_log_meta","init",1],"pid":"<0.349.0>"}]
application.rabbitmq-ha.log: [1730293616.568897252, {"time":"2024-10-30 14:06:56.568604+01:00","level":"notice","msg":"WAL: ra_coordination_log_wal init, open tbls: ra_coordination_log_open_mem_tables, closed tbls: ra_coordination_log_closed_mem_tables","domain":"ra","file":"src/ra_log_wal.erl","line":250,"mfa":["ra_log_wal","init",1],"pid":"<0.354.0>"}]
application.rabbitmq-ha.log: [1730293616.580743212, {"time":"2024-10-30 14:06:56.580380+01:00","level":"info","msg":"\n Starting RabbitMQ 3.12.12 on Erlang 25.3.2.8 [jit]\n Copyright (c) 2007-2023 Broadcom Inc and/or its subsidiaries\n Licensed under the MPL 2.0. Website: https://rabbitmq.com","domain":"rabbitmq.prelaunch","file":"rabbit.erl","line":875,"mfa":["rabbit","start",2],"pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293616.591481837, {"stream":"stdout","logtag":"F"}]
application.rabbitmq-ha.log: [1730293616.591507650, {"stream":"stdout","logtag":"F","log":"  ##  ##      RabbitMQ 3.12.12"}]
application.rabbitmq-ha.log: [1730293616.591516350, {"stream":"stdout","logtag":"F","log":"  ##  ##"}]
application.rabbitmq-ha.log: [1730293616.591524967, {"stream":"stdout","logtag":"F","log":"  ##########  Copyright (c) 2007-2023 Broadcom Inc and/or its subsidiaries"}]
application.rabbitmq-ha.log: [1730293616.591532899, {"stream":"stdout","logtag":"F","log":"  ######  ##"}]
application.rabbitmq-ha.log: [1730293616.591540762, {"stream":"stdout","logtag":"F","log":"  ##########  Licensed under the MPL 2.0. Website: https://rabbitmq.com"}]
application.rabbitmq-ha.log: [1730293616.591547823, {"stream":"stdout","logtag":"F"}]
application.rabbitmq-ha.log: [1730293616.591555209, {"stream":"stdout","logtag":"F","log":"  Erlang:      25.3.2.8 [jit]"}]
application.rabbitmq-ha.log: [1730293616.591562888, {"stream":"stdout","logtag":"F","log":"  TLS Library: OpenSSL - OpenSSL 3.1.4 24 Oct 2023"}]
application.rabbitmq-ha.log: [1730293616.591571070, {"stream":"stdout","logtag":"F","log":"  Release series support status: supported"}]
application.rabbitmq-ha.log: [1730293616.591578061, {"stream":"stdout","logtag":"F"}]
application.rabbitmq-ha.log: [1730293616.591585430, {"stream":"stdout","logtag":"F","log":"  Doc guides:  https://rabbitmq.com/documentation.html"}]
application.rabbitmq-ha.log: [1730293616.591593086, {"stream":"stdout","logtag":"F","log":"  Support:     https://rabbitmq.com/contact.html"}]
application.rabbitmq-ha.log: [1730293616.591600507, {"stream":"stdout","logtag":"F","log":"  Tutorials:   https://rabbitmq.com/getstarted.html"}]
application.rabbitmq-ha.log: [1730293616.591607790, {"stream":"stdout","logtag":"F","log":"  Monitoring:  https://rabbitmq.com/monitoring.html"}]
application.rabbitmq-ha.log: [1730293616.591614684, {"stream":"stdout","logtag":"F"}]
application.rabbitmq-ha.log: [1730293616.591622039, {"stream":"stdout","logtag":"F","log":"  Logs: <stdout>"}]
application.rabbitmq-ha.log: [1730293616.591628970, {"stream":"stdout","logtag":"F"}]
application.rabbitmq-ha.log: [1730293616.591636433, {"stream":"stdout","logtag":"F","log":"  Config file(s): /etc/rabbitmq/rabbitmq.conf"}]
application.rabbitmq-ha.log: [1730293616.591643434, {"stream":"stdout","logtag":"F"}]
application.rabbitmq-ha.log: [1730293616.591848214, {"stream":"stdout","logtag":"F","log":"  Starting broker...{\"time\":\"2024-10-30 14:06:56.591613+01:00\",\"level\":\"info\",\"msg\":\"\\n node           : rabbit@eisrabbitmq-rabbitmq-ha-0.eisrabbitmq-rabbitmq-ha-discovery.edison-system.svc.cluster.local\\n home dir       : /var/lib/rabbitmq\\n config file(s) : /etc/rabbitmq/rabbitmq.conf\\n cookie hash    : 2AnXraASrd+rMV02+02Jew==\\n log(s)         : <stdout>\\n data dir       : /var/lib/rabbitmq/mnesia/rabbit@eisrabbitmq-rabbitmq-ha-0.eisrabbitmq-rabbitmq-ha-discovery.edison-system.svc.cluster.local\",\"domain\":\"rabbitmq\",\"file\":\"rabbit.erl\",\"line\":1328,\"mfa\":[\"rabbit\",\"log_banner\",0],\"pid\":\"<0.230.0>\"}"}]
application.rabbitmq-ha.log: [1730293629.448115372, {"time":"2024-10-30 14:07:09.447576+01:00","level":"info","msg":"Running boot step pre_boot defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.448167259, {"time":"2024-10-30 14:07:09.447747+01:00","level":"info","msg":"Running boot step rabbit_global_counters defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.448773474, {"time":"2024-10-30 14:07:09.448512+01:00","level":"info","msg":"Running boot step rabbit_osiris_metrics defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.448978268, {"time":"2024-10-30 14:07:09.448762+01:00","level":"info","msg":"Running boot step rabbit_core_metrics defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.453683826, {"time":"2024-10-30 14:07:09.453156+01:00","level":"info","msg":"Running boot step rabbit_alarm defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.465169533, {"time":"2024-10-30 14:07:09.463814+01:00","level":"info","msg":"Memory high watermark set to 511 MiB (536661196 bytes) of 853 MiB (894435328 bytes) total","domain":"rabbitmq","pid":"<0.373.0>"}]
application.rabbitmq-ha.log: [1730293629.477410473, {"time":"2024-10-30 14:07:09.476930+01:00","level":"info","msg":"Enabling free disk space monitoring (disk free space: 146377420800, total memory: 894435328)","domain":"rabbitmq","pid":"<0.375.0>"}]
application.rabbitmq-ha.log: [1730293629.477488927, {"time":"2024-10-30 14:07:09.477070+01:00","level":"info","msg":"Disk free limit set to 50MB","domain":"rabbitmq","pid":"<0.375.0>"}]
application.rabbitmq-ha.log: [1730293629.549075062, {"time":"2024-10-30 14:07:09.548428+01:00","level":"info","msg":"Running boot step code_server_cache defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.549140468, {"time":"2024-10-30 14:07:09.548783+01:00","level":"info","msg":"Running boot step file_handle_cache defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.549701770, {"time":"2024-10-30 14:07:09.549358+01:00","level":"info","msg":"Limiting to approx 1048479 file handles (943629 sockets)","pid":"<0.378.0>"}]
application.rabbitmq-ha.log: [1730293629.550065115, {"time":"2024-10-30 14:07:09.549653+01:00","level":"info","msg":"FHC read buffering: OFF","domain":"rabbitmq","file":"rabbit.erl","line":1615,"mfa":["rabbit","ensure_working_fhc",0],"pid":"<0.379.0>"}]
application.rabbitmq-ha.log: [1730293629.550106743, {"time":"2024-10-30 14:07:09.549798+01:00","level":"info","msg":"FHC write buffering: ON","domain":"rabbitmq","file":"rabbit.erl","line":1617,"mfa":["rabbit","ensure_working_fhc",0],"pid":"<0.379.0>"}]
application.rabbitmq-ha.log: [1730293629.551033001, {"time":"2024-10-30 14:07:09.550570+01:00","level":"info","msg":"Running boot step worker_pool defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.551364539, {"time":"2024-10-30 14:07:09.550739+01:00","level":"info","msg":"Will use 32 processes for default worker pool","domain":"rabbitmq","pid":"<0.356.0>"}]
application.rabbitmq-ha.log: [1730293629.551471195, {"time":"2024-10-30 14:07:09.550864+01:00","level":"info","msg":"Starting worker pool 'worker_pool' with 32 processes in it","domain":"rabbitmq","pid":"<0.356.0>"}]
application.rabbitmq-ha.log: [1730293629.553750987, {"time":"2024-10-30 14:07:09.553436+01:00","level":"info","msg":"Running boot step database defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.557613251, {"time":"2024-10-30 14:07:09.556707+01:00","level":"info","msg":"Waiting for Mnesia tables for 30000 ms, 9 retries left","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.571474065, {"time":"2024-10-30 14:07:09.571055+01:00","level":"info","msg":"Successfully synced tables from a peer","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.574812646, {"time":"2024-10-30 14:07:09.574528+01:00","level":"info","msg":"Waiting for Mnesia tables for 30000 ms, 9 retries left","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.574991663, {"time":"2024-10-30 14:07:09.574796+01:00","level":"info","msg":"Successfully synced tables from a peer","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.575483325, {"time":"2024-10-30 14:07:09.575143+01:00","level":"info","msg":"Will register with peer discovery backend rabbit_peer_discovery_k8s","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.578453239, {"time":"2024-10-30 14:07:09.578139+01:00","level":"warning","msg":"Description: \"Server authenticity is not verified since certificate path validation is not enabled\"\n     Reason: \"The option {verify, verify_peer} and one of the options 'cacertfile' or 'cacerts' are required to enable this.\"\n","depth":20,"file":"ssl.erl","line":2922,"mfa":["ssl","maybe_client_warn_no_verify",2],"pid":"<0.418.0>"}]
application.rabbitmq-ha.log: [1730293629.593934126, {"time":"2024-10-30 14:07:09.593673+01:00","level":"info","msg":"Running boot step tracking_metadata_store defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.594092402, {"time":"2024-10-30 14:07:09.593863+01:00","level":"info","msg":"Setting up a table for connection tracking on this node: tracked_connection","domain":"rabbitmq","pid":"<0.426.0>"}]
application.rabbitmq-ha.log: [1730293629.594193018, {"time":"2024-10-30 14:07:09.593997+01:00","level":"info","msg":"Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhost","domain":"rabbitmq","pid":"<0.426.0>"}]
application.rabbitmq-ha.log: [1730293629.594319948, {"time":"2024-10-30 14:07:09.594131+01:00","level":"info","msg":"Setting up a table for per-user connection counting on this node: tracked_connection_per_user","domain":"rabbitmq","pid":"<0.426.0>"}]
application.rabbitmq-ha.log: [1730293629.594409672, {"time":"2024-10-30 14:07:09.594244+01:00","level":"info","msg":"Setting up a table for channel tracking on this node: tracked_channel","domain":"rabbitmq","pid":"<0.426.0>"}]
application.rabbitmq-ha.log: [1730293629.594526925, {"time":"2024-10-30 14:07:09.594361+01:00","level":"info","msg":"Setting up a table for channel tracking on this node: tracked_channel_per_user","domain":"rabbitmq","pid":"<0.426.0>"}]
application.rabbitmq-ha.log: [1730293629.594758656, {"time":"2024-10-30 14:07:09.594506+01:00","level":"info","msg":"Running boot step networking_metadata_store defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.594887755, {"time":"2024-10-30 14:07:09.594726+01:00","level":"info","msg":"Running boot step feature_flags defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.595120161, {"time":"2024-10-30 14:07:09.594961+01:00","level":"info","msg":"Running boot step codec_correctness_check defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.595226317, {"time":"2024-10-30 14:07:09.595042+01:00","level":"info","msg":"Running boot step rabbit_exchange_type_consistent_hash_metadata_store defined by app rabbitmq_consistent_hash_exchange","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.595777599, {"time":"2024-10-30 14:07:09.595587+01:00","level":"info","msg":"Waiting for Mnesia tables for 30000 ms, 0 retries left","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.595885941, {"time":"2024-10-30 14:07:09.595734+01:00","level":"info","msg":"Successfully synced tables from a peer","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.596157509, {"time":"2024-10-30 14:07:09.595943+01:00","level":"info","msg":"Running boot step external_infrastructure defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.596228757, {"time":"2024-10-30 14:07:09.596040+01:00","level":"info","msg":"Running boot step rabbit_event defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.596494822, {"time":"2024-10-30 14:07:09.596299+01:00","level":"info","msg":"Running boot step rabbit_registry defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.596708176, {"time":"2024-10-30 14:07:09.596495+01:00","level":"info","msg":"Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.596907119, {"time":"2024-10-30 14:07:09.596697+01:00","level":"info","msg":"Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.597029591, {"time":"2024-10-30 14:07:09.596853+01:00","level":"info","msg":"Running boot step rabbit_auth_mechanism_plain defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.597156815, {"time":"2024-10-30 14:07:09.596979+01:00","level":"info","msg":"Running boot step rabbit_exchange_type_direct defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.597295426, {"time":"2024-10-30 14:07:09.597098+01:00","level":"info","msg":"Running boot step rabbit_exchange_type_fanout defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.597390175, {"time":"2024-10-30 14:07:09.597224+01:00","level":"info","msg":"Running boot step rabbit_exchange_type_headers defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.597474306, {"time":"2024-10-30 14:07:09.597329+01:00","level":"info","msg":"Running boot step rabbit_exchange_type_topic defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.597582444, {"time":"2024-10-30 14:07:09.597427+01:00","level":"info","msg":"Running boot step rabbit_mirror_queue_mode_all defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.597710369, {"time":"2024-10-30 14:07:09.597526+01:00","level":"info","msg":"Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.597798945, {"time":"2024-10-30 14:07:09.597621+01:00","level":"info","msg":"Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.597886883, {"time":"2024-10-30 14:07:09.597743+01:00","level":"info","msg":"Running boot step rabbit_priority_queue defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.597939915, {"time":"2024-10-30 14:07:09.597812+01:00","level":"info","msg":"Priority queues enabled, real BQ is rabbit_variable_queue","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.598076195, {"time":"2024-10-30 14:07:09.597913+01:00","level":"info","msg":"Running boot step rabbit_queue_location_client_local defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.598165893, {"time":"2024-10-30 14:07:09.598018+01:00","level":"info","msg":"Running boot step rabbit_queue_location_min_masters defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.598292386, {"time":"2024-10-30 14:07:09.598120+01:00","level":"info","msg":"Running boot step rabbit_queue_location_random defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.598470268, {"time":"2024-10-30 14:07:09.598237+01:00","level":"info","msg":"Running boot step rabbit_exchange_type_consistent_hash_registry defined by app rabbitmq_consistent_hash_exchange","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.598548169, {"time":"2024-10-30 14:07:09.598359+01:00","level":"info","msg":"Running boot step kernel_ready defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.598569533, {"time":"2024-10-30 14:07:09.598446+01:00","level":"info","msg":"Running boot step rabbit_sysmon_minder defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.598926193, {"time":"2024-10-30 14:07:09.598743+01:00","level":"info","msg":"Running boot step rabbit_epmd_monitor defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.612124117, {"time":"2024-10-30 14:07:09.600078+01:00","level":"info","msg":"epmd monitor knows us, inter-node communication (distribution) port: 25672","domain":"rabbitmq","pid":"<0.437.0>"}]
application.rabbitmq-ha.log: [1730293629.612156231, {"time":"2024-10-30 14:07:09.600280+01:00","level":"info","msg":"Running boot step guid_generator defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.616150798, {"time":"2024-10-30 14:07:09.615560+01:00","level":"info","msg":"Running boot step rabbit_node_monitor defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.616537128, {"time":"2024-10-30 14:07:09.616171+01:00","level":"info","msg":"Starting rabbit_node_monitor (in autoheal mode)","domain":"rabbitmq","pid":"<0.441.0>"}]
application.rabbitmq-ha.log: [1730293629.616685296, {"time":"2024-10-30 14:07:09.616456+01:00","level":"info","msg":"Running boot step delegate_sup defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.617642207, {"time":"2024-10-30 14:07:09.617427+01:00","level":"info","msg":"Running boot step rabbit_memory_monitor defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.618000251, {"time":"2024-10-30 14:07:09.617817+01:00","level":"info","msg":"Running boot step rabbit_fifo_dlx_sup defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.618168332, {"time":"2024-10-30 14:07:09.617957+01:00","level":"info","msg":"Running boot step core_initialized defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.618199228, {"time":"2024-10-30 14:07:09.618076+01:00","level":"info","msg":"Running boot step rabbit_channel_tracking_handler defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.618361089, {"time":"2024-10-30 14:07:09.618167+01:00","level":"info","msg":"Running boot step rabbit_connection_tracking_handler defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.618495427, {"time":"2024-10-30 14:07:09.618265+01:00","level":"info","msg":"Running boot step rabbit_definitions_hashing defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.618712188, {"time":"2024-10-30 14:07:09.618460+01:00","level":"info","msg":"Running boot step rabbit_exchange_parameters defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.619236409, {"time":"2024-10-30 14:07:09.618951+01:00","level":"info","msg":"Running boot step rabbit_mirror_queue_misc defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.652722906, {"time":"2024-10-30 14:07:09.619673+01:00","level":"info","msg":"Running boot step rabbit_policies defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.659583139, {"time":"2024-10-30 14:07:09.652626+01:00","level":"info","msg":"Running boot step rabbit_policy defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.659615637, {"time":"2024-10-30 14:07:09.652934+01:00","level":"info","msg":"Running boot step rabbit_queue_location_validator defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.659625408, {"time":"2024-10-30 14:07:09.653143+01:00","level":"info","msg":"Running boot step rabbit_quorum_memory_manager defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.659633447, {"time":"2024-10-30 14:07:09.653331+01:00","level":"info","msg":"Running boot step rabbit_stream_coordinator defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.659641614, {"time":"2024-10-30 14:07:09.653615+01:00","level":"info","msg":"Running boot step rabbit_vhost_limit defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.659650087, {"time":"2024-10-30 14:07:09.653788+01:00","level":"info","msg":"Running boot step rabbit_federation_parameters defined by app rabbitmq_federation","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.659658056, {"time":"2024-10-30 14:07:09.654107+01:00","level":"info","msg":"Running boot step rabbit_federation_supervisor defined by app rabbitmq_federation","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.659665511, {"time":"2024-10-30 14:07:09.655798+01:00","level":"info","msg":"Running boot step rabbit_federation_queue defined by app rabbitmq_federation","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.659683779, {"time":"2024-10-30 14:07:09.656167+01:00","level":"info","msg":"Running boot step rabbit_federation_upstream_exchange defined by app rabbitmq_federation","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.659692329, {"time":"2024-10-30 14:07:09.656312+01:00","level":"info","msg":"Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.659699931, {"time":"2024-10-30 14:07:09.656411+01:00","level":"info","msg":"Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.659725828, {"time":"2024-10-30 14:07:09.656497+01:00","level":"info","msg":"Management plugin: using rates mode 'basic'","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.659734451, {"time":"2024-10-30 14:07:09.657726+01:00","level":"info","msg":"Running boot step rabbit_shovel_parameters defined by app rabbitmq_shovel","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.659741990, {"time":"2024-10-30 14:07:09.657918+01:00","level":"info","msg":"Running boot step recovery defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293629.662859485, {"time":"2024-10-30 14:07:09.662563+01:00","level":"info","msg":"Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@eisrabbitmq-rabbitmq-ha-0.eisrabbitmq-rabbitmq-ha-discovery.edison-system.svc.cluster.local/msg_stores/vhosts/3DLGUHJNW6IHWXY9I80NUUTRQ' for vhost 'VhostAnonymized' exists","domain":"rabbitmq","pid":"<0.490.0>"}]
application.rabbitmq-ha.log: [1730293629.673003404, {"time":"2024-10-30 14:07:09.672726+01:00","level":"info","msg":"Starting message stores for vhost 'VhostAnonymized'","domain":"rabbitmq","pid":"<0.490.0>"}]
application.rabbitmq-ha.log: [1730293629.673372482, {"time":"2024-10-30 14:07:09.673111+01:00","level":"info","msg":"Message store \"3DLGUHJNW6IHWXY9I80NUUTRQ/msg_store_transient\": using rabbit_msg_store_ets_index to provide index","domain":"rabbitmq","pid":"<0.499.0>"}]
application.rabbitmq-ha.log: [1730293629.676697946, {"time":"2024-10-30 14:07:09.676452+01:00","level":"info","msg":"Started message store of type transient for vhost 'VhostAnonymized'","domain":"rabbitmq","pid":"<0.490.0>"}]
application.rabbitmq-ha.log: [1730293629.677170964, {"time":"2024-10-30 14:07:09.676879+01:00","level":"info","msg":"Message store \"3DLGUHJNW6IHWXY9I80NUUTRQ/msg_store_persistent\": using rabbit_msg_store_ets_index to provide index","domain":"rabbitmq","pid":"<0.503.0>"}]
application.rabbitmq-ha.log: [1730293629.684419239, {"time":"2024-10-30 14:07:09.684160+01:00","level":"info","msg":"Started message store of type persistent for vhost 'VhostAnonymized'","domain":"rabbitmq","pid":"<0.490.0>"}]
application.rabbitmq-ha.log: [1730293630.051789967, {"time":"2024-10-30 14:07:10.044986+01:00","level":"info","msg":"Recovering 50 queues of type rabbit_classic_queue took 378ms","domain":"rabbitmq","pid":"<0.490.0>"}]
application.rabbitmq-ha.log: [1730293630.051834281, {"time":"2024-10-30 14:07:10.045174+01:00","level":"info","msg":"Recovering 0 queues of type rabbit_quorum_queue took 0ms","domain":"rabbitmq","pid":"<0.490.0>"}]
application.rabbitmq-ha.log: [1730293630.051844180, {"time":"2024-10-30 14:07:10.045263+01:00","level":"info","msg":"Recovering 0 queues of type rabbit_stream_queue took 0ms","domain":"rabbitmq","pid":"<0.490.0>"}]
application.rabbitmq-ha.log: [1730293630.148343814, {"time":"2024-10-30 14:07:10.147866+01:00","level":"info","msg":"Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@eisrabbitmq-rabbitmq-ha-0.eisrabbitmq-rabbitmq-ha-discovery.edison-system.svc.cluster.local/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists","domain":"rabbitmq","pid":"<0.792.0>"}]
application.rabbitmq-ha.log: [1730293630.159314988, {"time":"2024-10-30 14:07:10.158766+01:00","level":"info","msg":"Starting message stores for vhost '/'","domain":"rabbitmq","pid":"<0.792.0>"}]
application.rabbitmq-ha.log: [1730293630.164203793, {"time":"2024-10-30 14:07:10.159317+01:00","level":"info","msg":"Message store \"628WB79CIFDYO9LJI6DKMI09L/msg_store_transient\": using rabbit_msg_store_ets_index to provide index","domain":"rabbitmq","pid":"<0.801.0>"}]
application.rabbitmq-ha.log: [1730293630.169120887, {"time":"2024-10-30 14:07:10.168685+01:00","level":"info","msg":"Started message store of type transient for vhost '/'","domain":"rabbitmq","pid":"<0.792.0>"}]
application.rabbitmq-ha.log: [1730293630.169582942, {"time":"2024-10-30 14:07:10.169218+01:00","level":"info","msg":"Message store \"628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent\": using rabbit_msg_store_ets_index to provide index","domain":"rabbitmq","pid":"<0.805.0>"}]
application.rabbitmq-ha.log: [1730293630.173571628, {"time":"2024-10-30 14:07:10.173259+01:00","level":"info","msg":"Started message store of type persistent for vhost '/'","domain":"rabbitmq","pid":"<0.792.0>"}]
application.rabbitmq-ha.log: [1730293630.179269512, {"time":"2024-10-30 14:07:10.178934+01:00","level":"info","msg":"Recovering 2 queues of type rabbit_classic_queue took 29ms","domain":"rabbitmq","pid":"<0.792.0>"}]
application.rabbitmq-ha.log: [1730293630.179289686, {"time":"2024-10-30 14:07:10.179094+01:00","level":"info","msg":"Recovering 0 queues of type rabbit_quorum_queue took 0ms","domain":"rabbitmq","pid":"<0.792.0>"}]
application.rabbitmq-ha.log: [1730293630.179372692, {"time":"2024-10-30 14:07:10.179172+01:00","level":"info","msg":"Recovering 0 queues of type rabbit_stream_queue took 0ms","domain":"rabbitmq","pid":"<0.792.0>"}]
application.rabbitmq-ha.log: [1730293630.262108958, {"time":"2024-10-30 14:07:10.261677+01:00","level":"info","msg":"Running boot step empty_db_check defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293630.262185463, {"time":"2024-10-30 14:07:10.261857+01:00","level":"info","msg":"Will not seed default virtual host and user: have definitions to load...","domain":"rabbitmq","file":"rabbit.erl","line":1065,"mfa":["rabbit","maybe_insert_default_data",0],"pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293630.262231123, {"time":"2024-10-30 14:07:10.262011+01:00","level":"info","msg":"Running boot step rabbit_observer_cli defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293630.262441382, {"time":"2024-10-30 14:07:10.262162+01:00","level":"info","msg":"Running boot step rabbit_looking_glass defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293630.262483721, {"time":"2024-10-30 14:07:10.262302+01:00","level":"info","msg":"Running boot step rabbit_core_metrics_gc defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293630.262865340, {"time":"2024-10-30 14:07:10.262666+01:00","level":"info","msg":"Running boot step background_gc defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293630.263288893, {"time":"2024-10-30 14:07:10.263039+01:00","level":"info","msg":"Running boot step routing_ready defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293630.263348823, {"time":"2024-10-30 14:07:10.263195+01:00","level":"info","msg":"Running boot step pre_flight defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293630.263407406, {"time":"2024-10-30 14:07:10.263263+01:00","level":"info","msg":"Running boot step notify_cluster defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293630.263481099, {"time":"2024-10-30 14:07:10.263333+01:00","level":"info","msg":"Running boot step networking defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293630.263565537, {"time":"2024-10-30 14:07:10.263408+01:00","level":"info","msg":"Running boot step definition_import_worker_pool defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293630.263723880, {"time":"2024-10-30 14:07:10.263514+01:00","level":"info","msg":"Starting worker pool 'definition_import_pool' with 32 processes in it","domain":"rabbitmq","pid":"<0.356.0>"}]
application.rabbitmq-ha.log: [1730293630.266881998, {"time":"2024-10-30 14:07:10.266592+01:00","level":"info","msg":"Running boot step cluster_name defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293630.266954209, {"time":"2024-10-30 14:07:10.266740+01:00","level":"info","msg":"Running boot step direct_client defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293630.267165730, {"time":"2024-10-30 14:07:10.266947+01:00","level":"info","msg":"Running boot step rabbit_federation_exchange defined by app rabbitmq_federation","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293630.267570578, {"time":"2024-10-30 14:07:10.267389+01:00","level":"info","msg":"Running boot step rabbit_maintenance_mode_state defined by app rabbit","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293630.267594009, {"time":"2024-10-30 14:07:10.267478+01:00","level":"info","msg":"Creating table rabbit_node_maintenance_states for maintenance mode status","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293630.268521162, {"time":"2024-10-30 14:07:10.268175+01:00","level":"info","msg":"Running boot step rabbit_management_load_definitions defined by app rabbitmq_management","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293630.268934901, {"time":"2024-10-30 14:07:10.268647+01:00","level":"info","msg":"Applying definitions from regular file at /etc/definitions/definitions.json","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293630.269230790, {"time":"2024-10-30 14:07:10.269022+01:00","level":"info","msg":"Applying definitions from file at '/etc/definitions/definitions.json'","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293630.269271597, {"time":"2024-10-30 14:07:10.269132+01:00","level":"info","msg":"Asked to import definitions. Acting user: rmq-internal","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293630.269547403, {"time":"2024-10-30 14:07:10.269282+01:00","level":"info","msg":"Importing concurrently 2 users...","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293630.271109939, {"time":"2024-10-30 14:07:10.270854+01:00","level":"info","msg":"Successfully changed password for user 'management'","domain":"rabbitmq","pid":"<0.858.0>"}]
application.rabbitmq-ha.log: [1730293630.271145983, {"time":"2024-10-30 14:07:10.270868+01:00","level":"info","msg":"Successfully changed password for user 'admin'","domain":"rabbitmq","pid":"<0.859.0>"}]
application.rabbitmq-ha.log: [1730293630.271246750, {"time":"2024-10-30 14:07:10.271013+01:00","level":"info","msg":"Successfully set user tags for user 'admin' to [administrator]","domain":"rabbitmq","pid":"<0.859.0>"}]
application.rabbitmq-ha.log: [1730293630.271260913, {"time":"2024-10-30 14:07:10.270988+01:00","level":"info","msg":"Successfully set user tags for user 'management' to [management]","domain":"rabbitmq","pid":"<0.858.0>"}]
application.rabbitmq-ha.log: [1730293630.271508293, {"time":"2024-10-30 14:07:10.271249+01:00","level":"info","msg":"Importing concurrently 1 vhosts...","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293630.274496161, {"time":"2024-10-30 14:07:10.274214+01:00","level":"info","msg":"Importing concurrently 1 permissions...","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293630.275763110, {"time":"2024-10-30 14:07:10.275455+01:00","level":"info","msg":"Successfully set permissions for user 'admin' in virtual host '/' to '.*', '.*', '.*'","domain":"rabbitmq","pid":"<0.858.0>"}]
application.rabbitmq-ha.log: [1730293630.758682909, {"time":"2024-10-30 14:07:10.758246+01:00","level":"info","msg":"Resetting node maintenance status","file":"rabbit.erl","line":969,"mfa":["rabbit","do_run_postlaunch_phase",1],"pid":"<0.912.0>"}]
application.rabbitmq-ha.log: [1730293631.049884297, {"time":"2024-10-30 14:07:11.049429+01:00","level":"info","msg":"Management plugin: HTTP (non-TLS) listener started on port 15672","domain":"rabbitmq","pid":"<0.971.0>"}]
application.rabbitmq-ha.log: [1730293631.049996357, {"time":"2024-10-30 14:07:11.049761+01:00","level":"info","msg":"Statistics database started.","domain":"rabbitmq","pid":"<0.999.0>"}]
application.rabbitmq-ha.log: [1730293631.050251462, {"time":"2024-10-30 14:07:11.049943+01:00","level":"info","msg":"Starting worker pool 'management_worker_pool' with 3 processes in it","domain":"rabbitmq","pid":"<0.998.0>"}]
application.rabbitmq-ha.log: [1730293631.051627001, {"time":"2024-10-30 14:07:11.051351+01:00","level":"info","msg":"rabbit_stomp: default user 'guest' enabled","domain":"rabbitmq","pid":"<0.1008.0>"}]
application.rabbitmq-ha.log: [1730293631.054438831, {"time":"2024-10-30 14:07:11.054128+01:00","level":"info","msg":"started STOMP TCP listener on [::]:61613","pid":"<0.1027.0>"}]
application.rabbitmq-ha.log: [1730293631.059905252, {"time":"2024-10-30 14:07:11.059576+01:00","level":"info","msg":"rabbit_web_stomp: listening for HTTP connections on 0.0.0.0:15674","domain":"rabbitmq.connection","pid":"<0.1030.0>"}]
application.rabbitmq-ha.log: [1730293631.062359368, {"time":"2024-10-30 14:07:11.061981+01:00","level":"info","msg":"Peer discovery: enabling node cleanup (will only log warnings). Check interval: 10 seconds.","domain":"rabbitmq.peer_discovery","file":"rabbit_peer_discovery_cleanup.erl","line":102,"mfa":["rabbit_peer_discovery_cleanup","init",0],"pid":"<0.1063.0>"}]
application.rabbitmq-ha.log: [1730293631.063789914, {"time":"2024-10-30 14:07:11.063523+01:00","level":"info","msg":"Ready to start client connection listeners","file":"rabbit.erl","line":996,"mfa":["rabbit","do_run_postlaunch_phase",1],"pid":"<0.912.0>"}]
application.rabbitmq-ha.log: [1730293631.068394346, {"time":"2024-10-30 14:07:11.068085+01:00","level":"info","msg":"started TCP listener on [::]:5672","pid":"<0.1093.0>"}]
application.rabbitmq-ha.log: [1730293631.073462652, {"time":"2024-10-30 14:07:11.073156+01:00","level":"info","msg":"started TLS (SSL) listener on [::]:5671","pid":"<0.1113.0>"}]
application.rabbitmq-ha.log: [1730293631.469825823, {"stream":"stdout","logtag":"F","log":" completed with 12 plugins."}]
application.rabbitmq-ha.log: [1730293631.470112905, {"time":"2024-10-30 14:07:11.469258+01:00","level":"info","msg":"Server startup complete; 12 plugins started.\n * rabbitmq_shovel_management\n * rabbitmq_peer_discovery_k8s\n * rabbitmq_federation_management\n * rabbitmq_federation\n * rabbitmq_consistent_hash_exchange\n * rabbitmq_peer_discovery_common\n * rabbitmq_shovel\n * rabbitmq_web_stomp\n * rabbitmq_stomp\n * rabbitmq_management\n * rabbitmq_management_agent\n * rabbitmq_web_dispatch","domain":"rabbitmq","file":"rabbit.erl","line":1175,"mfa":["rabbit","log_broker_started",1],"pid":"<0.912.0>"}]
application.rabbitmq-ha.log: [1730293631.638511107, {"time":"2024-10-30 14:07:11.562223+01:00","level":"info","msg":"Time to start RabbitMQ: 30184968 us","file":"rabbit.erl","line":385,"mfa":["rabbit","start_it",1],"pid":"<0.9.0>"}]
application.rabbitmq-ha.log: [1730293631.801193827, {"time":"2024-10-30 14:07:11.799283+01:00","level":"info","msg":"accepting AMQP connection <0.1116.0> (198.18.95.213:38404 -> 198.18.95.195:5672)","domain":"rabbitmq.connection","pid":"<0.1116.0>"}]
application.rabbitmq-ha.log: [1730293631.806690992, {"time":"2024-10-30 14:07:11.802171+01:00","level":"info","msg":"connection <0.1116.0> (198.18.95.213:38404 -> 198.18.95.195:5672) has a client-provided name: rabbitConnectionFactory#3130ca39:281599","domain":"rabbitmq.connection","pid":"<0.1116.0>"}]
application.rabbitmq-ha.log: [1730293631.814385894, {"time":"2024-10-30 14:07:11.813897+01:00","level":"info","msg":"connection <0.1116.0> (198.18.95.213:38404 -> 198.18.95.195:5672 - rabbitConnectionFactory#3130ca39:281599): user 'testuser' authenticated and granted access to vhost '/'","domain":"rabbitmq.connection","pid":"<0.1116.0>"}]
application.rabbitmq-ha.log: [1730293632.234191169, {"time":"2024-10-30 14:07:12.219898+01:00","level":"info","msg":"accepting AMQP connection <0.1131.0> (198.18.95.203:53764 -> 198.18.95.195:5672)","domain":"rabbitmq.connection","pid":"<0.1131.0>"}]
application.rabbitmq-ha.log: [1730293632.234228462, {"time":"2024-10-30 14:07:12.226567+01:00","level":"info","msg":"connection <0.1131.0> (198.18.95.203:53764 -> 198.18.95.195:5672) has a client-provided name: rabbitConnectionFactory#349e6ca8:2832918","domain":"rabbitmq.connection","pid":"<0.1131.0>"}]
application.rabbitmq-ha.log: [1730293632.234239542, {"time":"2024-10-30 14:07:12.228087+01:00","level":"info","msg":"connection <0.1131.0> (198.18.95.203:53764 -> 198.18.95.195:5672 - rabbitConnectionFactory#349e6ca8:2832918): user 'AnonymizedUser3' authenticated and granted access to vhost '/'","domain":"rabbitmq.connection","pid":"<0.1131.0>"}]
application.rabbitmq-ha.log: [1730293632.471453980, {"time":"2024-10-30 14:07:12.471017+01:00","level":"info","msg":"accepting AMQP connection <0.1172.0> (198.18.95.197:38622 -> 198.18.95.195:5672)","domain":"rabbitmq.connection","pid":"<0.1172.0>"}]
application.rabbitmq-ha.log: [1730293632.473568243, {"time":"2024-10-30 14:07:12.473237+01:00","level":"info","msg":"connection <0.1172.0> (198.18.95.197:38622 -> 198.18.95.195:5672) has a client-provided name: rabbitConnectionFactory#65e92505:4713893","domain":"rabbitmq.connection","pid":"<0.1172.0>"}]
application.rabbitmq-ha.log: [1730293632.474710756, {"time":"2024-10-30 14:07:12.474469+01:00","level":"info","msg":"connection <0.1172.0> (198.18.95.197:38622 -> 198.18.95.195:5672 - rabbitConnectionFactory#65e92505:4713893): user 'AnonymizedUser4' authenticated and granted access to vhost '/'","domain":"rabbitmq.connection","pid":"<0.1172.0>"}]
application.rabbitmq-ha.log: [1730293634.308960356, {"time":"2024-10-30 14:07:14.308455+01:00","level":"info","msg":"accepting AMQP connection <0.1241.0> (198.18.95.223:47562 -> 198.18.95.195:5672)","domain":"rabbitmq.connection","pid":"<0.1241.0>"}]
application.rabbitmq-ha.log: [1730293634.311266776, {"time":"2024-10-30 14:07:14.310972+01:00","level":"info","msg":"connection <0.1241.0> (198.18.95.223:47562 -> 198.18.95.195:5672) has a client-provided name: rabbitConnectionFactory#53fd59d4:281755","domain":"rabbitmq.connection","pid":"<0.1241.0>"}]
application.rabbitmq-ha.log: [1730293634.312830869, {"time":"2024-10-30 14:07:14.312542+01:00","level":"info","msg":"connection <0.1241.0> (198.18.95.223:47562 -> 198.18.95.195:5672 - rabbitConnectionFactory#53fd59d4:281755): user 'eisdicomexportserviceuser' authenticated and granted access to vhost '/'","domain":"rabbitmq.connection","pid":"<0.1241.0>"}]
application.rabbitmq-ha.log: [1730293650.917450331, {"time":"2024-10-30 14:07:30.916239+01:00","level":"info","msg":"Web STOMP: file handle cache use is enabled","file":"rabbit_web_stomp_handler.erl","line":90,"mfa":["rabbit_web_stomp_handler","init",2],"pid":"<0.1260.0>"}]
application.rabbitmq-ha.log: [1730293650.929526043, {"time":"2024-10-30 14:07:30.929016+01:00","level":"warning","msg":"AMQP 0-9-1 client call timeout was 70000 ms, is updated to a safe effective value of 130000 ms","pid":"<0.1259.0>"}]
application.rabbitmq-ha.log: [1730293691.462156172, {"time":"2024-10-30 14:08:11.461680+01:00","level":"info","msg":"accepting AMQP connection <0.1306.0> (198.18.170.2:61574 -> 198.18.95.195:5672)","domain":"rabbitmq.connection","pid":"<0.1306.0>"}]
application.rabbitmq-ha.log: [1730293691.463202914, {"time":"2024-10-30 14:08:11.462436+01:00","level":"info","msg":"accepting AMQP connection <0.1309.0> (198.18.170.2:32264 -> 198.18.95.195:5672)","domain":"rabbitmq.connection","pid":"<0.1309.0>"}]
application.rabbitmq-ha.log: [1730293691.545219851, {"time":"2024-10-30 14:08:11.544743+01:00","level":"info","msg":"accepting AMQP connection <0.1312.0> (198.18.170.2:51625 -> 198.18.95.195:5672)","domain":"rabbitmq.connection","pid":"<0.1312.0>"}]
application.rabbitmq-ha.log: [1730293691.545744575, {"time":"2024-10-30 14:08:11.544985+01:00","level":"info","msg":"accepting AMQP connection <0.1315.0> (198.18.170.2:12252 -> 198.18.95.195:5672)","domain":"rabbitmq.connection","pid":"<0.1315.0>"}]
application.rabbitmq-ha.log: [1730293691.546066194, {"time":"2024-10-30 14:08:11.545348+01:00","level":"info","msg":"accepting AMQP connection <0.1318.0> (198.18.95.222:50762 -> 198.18.95.195:5672)","domain":"rabbitmq.connection","pid":"<0.1318.0>"}]
application.rabbitmq-ha.log: [1730293691.547457207, {"time":"2024-10-30 14:08:11.546611+01:00","level":"info","msg":"connection <0.1306.0> (198.18.170.2:61574 -> 198.18.95.195:5672): user 'AnonymizedUser1' authenticated and granted access to vhost 'VhostAnonymized'","domain":"rabbitmq.connection","pid":"<0.1306.0>"}]
application.rabbitmq-ha.log: [1730293691.643561932, {"time":"2024-10-30 14:08:11.548971+01:00","level":"info","msg":"connection <0.1318.0> (198.18.95.222:50762 -> 198.18.95.195:5672): user 'AnonymizedUser2' authenticated and granted access to vhost '/'","domain":"rabbitmq.connection","pid":"<0.1318.0>"}]
application.rabbitmq-ha.log: [1730293691.649016324, {"time":"2024-10-30 14:08:11.644957+01:00","level":"info","msg":"connection <0.1309.0> (198.18.170.2:32264 -> 198.18.95.195:5672): user 'AnonymizedUser1' authenticated and granted access to vhost '/'","domain":"rabbitmq.connection","pid":"<0.1309.0>"}]
application.rabbitmq-ha.log: [1730293691.649075015, {"time":"2024-10-30 14:08:11.646957+01:00","level":"info","msg":"connection <0.1312.0> (198.18.170.2:51625 -> 198.18.95.195:5672): user 'AnonymizedUser1' authenticated and granted access to vhost '/'","domain":"rabbitmq.connection","pid":"<0.1312.0>"}]
application.rabbitmq-ha.log: [1730293691.649090373, {"time":"2024-10-30 14:08:11.647572+01:00","level":"info","msg":"connection <0.1315.0> (198.18.170.2:12252 -> 198.18.95.195:5672): user 'AnonymizedUser1' authenticated and granted access to vhost 'VhostAnonymized'","domain":"rabbitmq.connection","pid":"<0.1315.0>"}]
application.rabbitmq-ha.log: [1730293923.144110619, {"time":"2024-10-30 14:12:03.143552+01:00","level":"notice","msg":"SIGTERM received - shutting down\n","pid":"<0.61.0>"}]
application.rabbitmq-ha.log: [1730293923.250299156, {"time":"2024-10-30 14:12:03.246970+01:00","level":"error","msg":"** Generic server <0.1263.0> terminating \n** Last message in was {'EXIT',<0.1259.0>,shutdown}\n** When Server state == {state,amqp_direct_connection,\n                         {state,\n                          'rabbit@eisrabbitmq-rabbitmq-ha-0.eisrabbitmq-rabbitmq-ha-discovery.edison-system.svc.cluster.local',\n                          {user,<<\"LimitedUser\">>,\n                           [administrator],\n                           [{rabbit_auth_backend_internal,\n                             #Fun<rabbit_auth_backend_internal.3.20918515>}]},\n                          <<\"VhostAnonymized\">>,\n                          {amqp_params_direct,<<\"LimitedUser\">>,\n                           {encrypted,\n                            <<\"/l31Id8GJUXjQ2/zWQm0R6jxn6hA3FYffz9qwyAAVUVzQKL/pUjOYCIbgCKfsuT9aaySlBEUkZmZwHZQdL8mqNhh9KvssQhS1I3tyaF0W0Y=\">>},\n                           <<\"VhostAnonymized\">>,\n                           'rabbit@eisrabbitmq-rabbitmq-ha-0.eisrabbitmq-rabbitmq-ha-discovery.edison-system.svc.cluster.local',\n                           {amqp_adapter_info,\n                            {198,18,95,195},\n                            15674,\n                            {198,18,95,202},\n                            55256,\n                            <<\"198.18.95.202:55256 -> 198.18.95.195:15674\">>,\n                            {'Web STOMP',\"1.2\"},\n                            [{channels,1},\n                             {channel_max,1},\n                             {frame_max,0},\n                             {client_properties,\n                              [{<<\"product\">>,longstr,<<\"STOMP client\">>}]},\n                             {ssl,false}]},\n                           []},\n                          {amqp_adapter_info,\n                           {198,18,95,195},\n                           15674,\n                           {198,18,95,202},\n                           55256,\n                           <<\"198.18.95.202:55256 -> 198.18.95.195:15674\">>,\n                           {'Web STOMP',\"1.2\"},\n                           [{channels,1},\n                            {channel_max,1},\n                            {frame_max,0},\n                            {client_properties,\n                             [{<<\"product\">>,longstr,<<\"STOMP client\">>}]},\n                            {ssl,false}]},\n                          <0.1268.0>,undefined,1730293650929},\n                         <0.1267.0>,\n                         {amqp_params_direct,<<\"LimitedUser\">>,\n                          {encrypted,\n                           <<\"/l31Id8GJUXjQ2/zWQm0R6jxn6hA3FYffz9qwyAAVUVzQKL/pUjOYCIbgCKfsuT9aaySlBEUkZmZwHZQdL8mqNhh9KvssQhS1I3tyaF0W0Y=\">>},\n                          <<\"VhostAnonymized\">>,\n                          'rabbit@eisrabbitmq-rabbitmq-ha-0.eisrabbitmq-rabbitmq-ha-discovery.edison-system.svc.cluster.local',\n                          {amqp_adapter_info,\n                           {198,18,95,195},\n                           15674,\n                           {198,18,95,202},\n                           55256,\n                           <<\"198.18.95.202:55256 -> 198.18.95.195:15674\">>,\n                           {'Web STOMP',\"1.2\"},\n                           [{channels,1},\n                            {channel_max,1},\n                            {frame_max,0},\n                            {client_properties,\n                             [{<<\"product\">>,longstr,<<\"STOMP client\">>}]},\n                            {ssl,false}]},\n                          []},\n                         0,\n                         [{<<\"capabilities\">>,table,\n                           [{<<\"publisher_confirms\">>,bool,true},\n                            {<<\"exchange_exchange_bindings\">>,bool,true},\n                            {<<\"basic.nack\">>,bool,true},\n                            {<<\"consumer_cancel_notify\">>,bool,true},\n                            {<<\"connection.blocked\">>,bool,true},\n                            {<<\"consumer_priorities\">>,bool,true},\n                            {<<\"authentication_failure_close\">>,bool,true},\n                            {<<\"per_consumer_qos\">>,bool,true},\n                            {<<\"direct_reply_to\">>,bool,true}]},\n                          {<<\"cluster_name\">>,longstr,\n                           <<\"rabbit@eisrabbitmq-rabbitmq-ha-0.eisrabbitmq-rabbitmq-ha-discovery.edison-system.svc.cluster.local\">>},\n                          {<<\"copyright\">>,longstr,\n                           <<\"Copyright (c) 2007-2023 Broadcom Inc and/or its subsidiaries\">>},\n                          {<<\"information\">>,longstr,\n                           <<\"Licensed under the MPL 2.0. Website: https://rabbitmq.com\">>},\n                          {<<\"platform\">>,longstr,<<\"Erlang/OTP 25.3.2.8\">>},\n                          {<<\"product\">>,longstr,<<\"RabbitMQ\">>},\n                          {<<\"version\">>,longstr,<<\"3.12.12\">>}],\n                         none,false}\n** Reason for termination ==\n** \"stopping because dependent process <0.1259.0> died: shutdown\"\n","domain":"otp","file":"gen_server.erl","line":1402,"mfa":["gen_server","error_info",8],"pid":"<0.1263.0>"}]
application.rabbitmq-ha.log: [1730293923.250365079, {"time":"2024-10-30 14:12:03.249684+01:00","level":"error","msg":"  crasher:\n    initial call: amqp_gen_connection:init/1\n    pid: <0.1263.0>\n    registered_name: []\n    exception exit: \"stopping because dependent process <0.1259.0> died: shutdown\"\n      in function  gen_server:handle_common_reply/8 (gen_server.erl, line 1241)\n    ancestors: [<0.1261.0>,amqp_sup,<0.921.0>]\n    message_queue_len: 0\n    messages: []\n    links: [<0.1261.0>]\n    dictionary: [{process_name,\n                      {amqp_gen_connection,\n                          <<\"198.18.95.202:55256 -> 198.18.95.195:15674\">>}},\n                  {gen_server_call_timeout,130000}]\n    trap_exit: true\n    status: running\n    heap_size: 4185\n    stack_size: 28\n    reductions: 51983\n  neighbours:\n","domain":"otp.sasl","file":"proc_lib.erl","line":539,"mfa":["proc_lib","crash_report",4],"pid":"<0.1263.0>"}]
application.rabbitmq-ha.log: [1730293923.251287305, {"time":"2024-10-30 14:12:03.250356+01:00","level":"info","msg":"stopped STOMP TCP listener on [::]:61613","pid":"<0.1027.0>"}]
application.rabbitmq-ha.log: [1730293923.251334655, {"time":"2024-10-30 14:12:03.250297+01:00","level":"error","msg":"    supervisor: {<0.1261.0>,amqp_connection_sup}\n    errorContext: child_terminated\n    reason: \"stopping because dependent process <0.1259.0> died: shutdown\"\n    offender: [{pid,<0.1263.0>},\n               {id,connection},\n               {mfargs,\n                   {amqp_gen_connection,start_link,\n                       [<0.1262.0>,\n                        {amqp_params_direct,<<\"LimitedUser\">>,\n                            {encrypted,\n                                <<\"/l31Id8GJUXjQ2/zWQm0R6jxn6hA3FYffz9qwyAAVUVzQKL/pUjOYCIbgCKfsuT9aaySlBEUkZmZwHZQdL8mqNhh9KvssQhS1I3tyaF0W0Y=\">>},\n                            <<\"VhostAnonymized\">>,\n                            'rabbit@eisrabbitmq-rabbitmq-ha-0.eisrabbitmq-rabbitmq-ha-discovery.edison-system.svc.cluster.local',\n                            {amqp_adapter_info,\n                                {198,18,95,195},\n                                15674,\n                                {198,18,95,202},\n                                55256,\n                                <<\"198.18.95.202:55256 -> 198.18.95.195:15674\">>,\n                                {'Web STOMP',\"1.2\"},\n                                [{channels,1},\n                                 {channel_max,1},\n                                 {frame_max,0},\n                                 {client_properties,\n                                     [{<<\"product\">>,longstr,\n                                       <<\"STOMP client\">>}]},\n                                 {ssl,false}]},\n                            []}]}},\n               {restart_type,transient},\n               {significant,true},\n               {shutdown,brutal_kill},\n               {child_type,worker}]\n","domain":"otp.sasl","file":"supervisor.erl","line":754,"mfa":["supervisor","do_restart",3],"pid":"<0.1261.0>"}]
application.rabbitmq-ha.log: [1730293923.343394649, {"time":"2024-10-30 14:12:03.251047+01:00","level":"error","msg":"    supervisor: {<0.1261.0>,amqp_connection_sup}\n    errorContext: shutdown\n    reason: reached_max_restart_intensity\n    offender: [{pid,<0.1263.0>},\n               {id,connection},\n               {mfargs,\n                   {amqp_gen_connection,start_link,\n                       [<0.1262.0>,\n                        {amqp_params_direct,<<\"LimitedUser\">>,\n                            {encrypted,\n                                <<\"/l31Id8GJUXjQ2/zWQm0R6jxn6hA3FYffz9qwyAAVUVzQKL/pUjOYCIbgCKfsuT9aaySlBEUkZmZwHZQdL8mqNhh9KvssQhS1I3tyaF0W0Y=\">>},\n                            <<\"VhostAnonymized\">>,\n                            'rabbit@eisrabbitmq-rabbitmq-ha-0.eisrabbitmq-rabbitmq-ha-discovery.edison-system.svc.cluster.local',\n                            {amqp_adapter_info,\n                                {198,18,95,195},\n                                15674,\n                                {198,18,95,202},\n                                55256,\n                                <<\"198.18.95.202:55256 -> 198.18.95.195:15674\">>,\n                                {'Web STOMP',\"1.2\"},\n                                [{channels,1},\n                                 {channel_max,1},\n                                 {frame_max,0},\n                                 {client_properties,\n                                     [{<<\"product\">>,longstr,\n                                       <<\"STOMP client\">>}]},\n                                 {ssl,false}]},\n                            []}]}},\n               {restart_type,transient},\n               {significant,true},\n               {shutdown,brutal_kill},\n               {child_type,worker}]\n","domain":"otp.sasl","file":"supervisor.erl","line":811,"mfa":["supervisor","restart",2],"pid":"<0.1261.0>"}]
application.rabbitmq-ha.log: [1730293923.548933825, {"time":"2024-10-30 14:12:03.460517+01:00","level":"warning","msg":"HTTP listener registry could not find context rabbitmq_management_tls","domain":"rabbitmq","pid":"<0.927.0>"}]
application.rabbitmq-ha.log: [1730293923.562371060, {"time":"2024-10-30 14:12:03.561907+01:00","level":"info","msg":"Will unregister with peer discovery backend rabbit_peer_discovery_k8s","domain":"rabbitmq","pid":"<0.230.0>"}]
application.rabbitmq-ha.log: [1730293923.563057951, {"time":"2024-10-30 14:12:03.562235+01:00","level":"info","msg":"stopped TLS (SSL) listener on [::]:5671","pid":"<0.1113.0>"}]
application.rabbitmq-ha.log: [1730293923.648043831, {"time":"2024-10-30 14:12:03.647530+01:00","level":"info","msg":"stopped TCP listener on [::]:5672","pid":"<0.1093.0>"}]
application.rabbitmq-ha.log: [1730293923.650056156, {"time":"2024-10-30 14:12:03.649317+01:00","level":"error","msg":"Error on AMQP connection <0.1306.0> (198.18.170.2:61574 -> 198.18.95.195:5672, vhost: 'VhostAnonymized', user: 'AnonymizedUser1', state: running), channel 0:\n operation none caused a connection exception connection_forced: \"broker forced connection closure with reason 'shutdown'\"","domain":"rabbitmq.connection","pid":"<0.1306.0>"}]
application.rabbitmq-ha.log: [1730293923.650179919, {"time":"2024-10-30 14:12:03.649315+01:00","level":"error","msg":"Error on AMQP connection <0.1309.0> (198.18.170.2:32264 -> 198.18.95.195:5672, vhost: '/', user: 'AnonymizedUser1', state: running), channel 0:\n operation none caused a connection exception connection_forced: \"broker forced connection closure with reason 'shutdown'\"","domain":"rabbitmq.connection","pid":"<0.1309.0>"}]
application.rabbitmq-ha.log: [1730293923.650228069, {"time":"2024-10-30 14:12:03.649329+01:00","level":"error","msg":"Error on AMQP connection <0.1116.0> (198.18.95.213:38404 -> 198.18.95.195:5672 - rabbitConnectionFactory#3130ca39:281599, vhost: '/', user: 'testuser', state: running), channel 0:\n operation none caused a connection exception connection_forced: \"broker forced connection closure with reason 'shutdown'\"","domain":"rabbitmq.connection","pid":"<0.1116.0>"}]
application.rabbitmq-ha.log: [1730293923.650238831, {"time":"2024-10-30 14:12:03.649314+01:00","level":"error","msg":"Error on AMQP connection <0.1241.0> (198.18.95.223:47562 -> 198.18.95.195:5672 - rabbitConnectionFactory#53fd59d4:281755, vhost: '/', user: 'eisdicomexportserviceuser', state: running), channel 0:\n operation none caused a connection exception connection_forced: \"broker forced connection closure with reason 'shutdown'\"","domain":"rabbitmq.connection","pid":"<0.1241.0>"}]
application.rabbitmq-ha.log: [1730293923.650248212, {"time":"2024-10-30 14:12:03.649662+01:00","level":"error","msg":"Error on AMQP connection <0.1318.0> (198.18.95.222:50762 -> 198.18.95.195:5672, vhost: '/', user: 'AnonymizedUser2', state: running), channel 0:\n operation none caused a connection exception connection_forced: \"broker forced connection closure with reason 'shutdown'\"","domain":"rabbitmq.connection","pid":"<0.1318.0>"}]
application.rabbitmq-ha.log: [1730293923.650378313, {"time":"2024-10-30 14:12:03.649645+01:00","level":"error","msg":"Error on AMQP connection <0.1131.0> (198.18.95.203:53764 -> 198.18.95.195:5672 - rabbitConnectionFactory#349e6ca8:2832918, vhost: '/', user: 'AnonymizedUser3', state: running), channel 0:\n operation none caused a connection exception connection_forced: \"broker forced connection closure with reason 'shutdown'\"","domain":"rabbitmq.connection","pid":"<0.1131.0>"}]
application.rabbitmq-ha.log: [1730293923.650393761, {"time":"2024-10-30 14:12:03.649606+01:00","level":"error","msg":"Error on AMQP connection <0.1315.0> (198.18.170.2:12252 -> 198.18.95.195:5672, vhost: 'VhostAnonymized', user: 'AnonymizedUser1', state: running), channel 0:\n operation none caused a connection exception connection_forced: \"broker forced connection closure with reason 'shutdown'\"","domain":"rabbitmq.connection","pid":"<0.1315.0>"}]
application.rabbitmq-ha.log: [1730293923.650438295, {"time":"2024-10-30 14:12:03.649564+01:00","level":"error","msg":"Error on AMQP connection <0.1172.0> (198.18.95.197:38622 -> 198.18.95.195:5672 - rabbitConnectionFactory#65e92505:4713893, vhost: '/', user: 'AnonymizedUser4', state: running), channel 0:\n operation none caused a connection exception connection_forced: \"broker forced connection closure with reason 'shutdown'\"","domain":"rabbitmq.connection","pid":"<0.1172.0>"}]
application.rabbitmq-ha.log: [1730293923.748763306, {"time":"2024-10-30 14:12:03.649593+01:00","level":"error","msg":"Error on AMQP connection <0.1312.0> (198.18.170.2:51625 -> 198.18.95.195:5672, vhost: '/', user: 'AnonymizedUser1', state: running), channel 0:\n operation none caused a connection exception connection_forced: \"broker forced connection closure with reason 'shutdown'\"","domain":"rabbitmq.connection","pid":"<0.1312.0>"}]
application.rabbitmq-ha.log: [1730293923.943440761, {"time":"2024-10-30 14:12:03.854449+01:00","level":"info","msg":"Closing all connections in vhost '/' on node 'rabbit@eisrabbitmq-rabbitmq-ha-0.eisrabbitmq-rabbitmq-ha-discovery.edison-system.svc.cluster.local' because the vhost is stopping","domain":"rabbitmq.connection","pid":"<0.1730.0>"}]
application.rabbitmq-ha.log: [1730293923.943515164, {"time":"2024-10-30 14:12:03.854449+01:00","level":"info","msg":"Closing all connections in vhost 'VhostAnonymized' on node 'rabbit@eisrabbitmq-rabbitmq-ha-0.eisrabbitmq-rabbitmq-ha-discovery.edison-system.svc.cluster.local' because the vhost is stopping","domain":"rabbitmq.connection","pid":"<0.1731.0>"}]
application.rabbitmq-ha.log: [1730293924.152051298, {"time":"2024-10-30 14:12:04.151564+01:00","level":"info","msg":"Stopping message store for directory '/var/lib/rabbitmq/mnesia/rabbit@eisrabbitmq-rabbitmq-ha-0.eisrabbitmq-rabbitmq-ha-discovery.edison-system.svc.cluster.local/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent'","domain":"rabbitmq","pid":"<0.805.0>"}]
application.rabbitmq-ha.log: [1730293924.152121763, {"time":"2024-10-30 14:12:04.151662+01:00","level":"info","msg":"Stopping message store for directory '/var/lib/rabbitmq/mnesia/rabbit@eisrabbitmq-rabbitmq-ha-0.eisrabbitmq-rabbitmq-ha-discovery.edison-system.svc.cluster.local/msg_stores/vhosts/3DLGUHJNW6IHWXY9I80NUUTRQ/msg_store_persistent'","domain":"rabbitmq","pid":"<0.503.0>"}]
application.rabbitmq-ha.log: [1730293924.267170385, {"time":"2024-10-30 14:12:04.266571+01:00","level":"info","msg":"Message store for directory '/var/lib/rabbitmq/mnesia/rabbit@eisrabbitmq-rabbitmq-ha-0.eisrabbitmq-rabbitmq-ha-discovery.edison-system.svc.cluster.local/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' is stopped","domain":"rabbitmq","pid":"<0.805.0>"}]
application.rabbitmq-ha.log: [1730293924.267245334, {"time":"2024-10-30 14:12:04.266908+01:00","level":"info","msg":"Stopping message store for directory '/var/lib/rabbitmq/mnesia/rabbit@eisrabbitmq-rabbitmq-ha-0.eisrabbitmq-rabbitmq-ha-discovery.edison-system.svc.cluster.local/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient'","domain":"rabbitmq","pid":"<0.801.0>"}]
application.rabbitmq-ha.log: [1730293924.287764406, {"time":"2024-10-30 14:12:04.287188+01:00","level":"info","msg":"Message store for directory '/var/lib/rabbitmq/mnesia/rabbit@eisrabbitmq-rabbitmq-ha-0.eisrabbitmq-rabbitmq-ha-discovery.edison-system.svc.cluster.local/msg_stores/vhosts/3DLGUHJNW6IHWXY9I80NUUTRQ/msg_store_persistent' is stopped","domain":"rabbitmq","pid":"<0.503.0>"}]
application.rabbitmq-ha.log: [1730293924.287795608, {"time":"2024-10-30 14:12:04.287469+01:00","level":"info","msg":"Stopping message store for directory '/var/lib/rabbitmq/mnesia/rabbit@eisrabbitmq-rabbitmq-ha-0.eisrabbitmq-rabbitmq-ha-discovery.edison-system.svc.cluster.local/msg_stores/vhosts/3DLGUHJNW6IHWXY9I80NUUTRQ/msg_store_transient'","domain":"rabbitmq","pid":"<0.499.0>"}]
application.rabbitmq-ha.log: [1730293924.444064522, {"time":"2024-10-30 14:12:04.443316+01:00","level":"info","msg":"Message store for directory '/var/lib/rabbitmq/mnesia/rabbit@eisrabbitmq-rabbitmq-ha-0.eisrabbitmq-rabbitmq-ha-discovery.edison-system.svc.cluster.local/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' is stopped","domain":"rabbitmq","pid":"<0.801.0>"}]
application.rabbitmq-ha.log: [1730293924.444118004, {"time":"2024-10-30 14:12:04.443478+01:00","level":"info","msg":"Message store for directory '/var/lib/rabbitmq/mnesia/rabbit@eisrabbitmq-rabbitmq-ha-0.eisrabbitmq-rabbitmq-ha-discovery.edison-system.svc.cluster.local/msg_stores/vhosts/3DLGUHJNW6IHWXY9I80NUUTRQ/msg_store_transient' is stopped","domain":"rabbitmq","pid":"<0.499.0>"}]
application.rabbitmq-ha.log: [1730293924.550336958, {"time":"2024-10-30 14:12:04.549790+01:00","level":"info","msg":"Management plugin: to stop collect_statistics.","domain":"rabbitmq","pid":"<0.432.0>"}]```
</details>


### Logs from node 2 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs
<details>

PASTE LOG HERE, BETWEEN BACKTICKS

</details>


### Logs from node 3 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs
<details>

PASTE LOG HERE, BETWEEN BACKTICKS

</details>


### rabbitmq.conf

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location
<details>

RabbitMQ configuration

## Ref: https://github.com/rabbitmq/rabbitmq-server/blob/master/docs/rabbitmq.conf.example

## Logging configuration
log.console = true
log.console.formatter = json
log.console.level = info

## Authentification

## Clustering
cluster_formation.peer_discovery_backend  = rabbit_peer_discovery_k8s
##Modified to be configurable for EIS K8s
cluster_formation.k8s.host = kubernetes.default

cluster_formation.k8s.address_type = hostname
# cluster_formation.k8s.address_type = ip

cluster_formation.node_cleanup.interval = 10
# Set to false if automatic cleanup of absent nodes is desired.
# This can be dangerous, see http://www.rabbitmq.com/cluster-formation.html#node-health-checks-and-cleanup.
cluster_formation.node_cleanup.only_log_warning = true
cluster_partition_handling = autoheal

## The default "guest" user is only permitted to access the server
## via a loopback interface (e.g. localhost)
#loopback_users.guest = false

management.load_definitions = /etc/definitions/definitions.json
management.path_prefix = /rabbitmq-admin

## Memory-based Flow Control options.
vm_memory_high_watermark.relative = 0.4
vm_memory_calculation_strategy = allocated

## Total memory available override.
total_memory_available_override_value = 400MiB

## Background garbage collection options.
background_gc_enabled = false
background_gc_target_interval = 60000

## Web STOMP Plugin
web_stomp.ws_frame = binary
web_stomp.cowboy_opts.max_keepalive = 10

</details>


### Steps to deploy RabbitMQ cluster

inside kubernetes using bitnami rabbitmq helm chart

### Steps to reproduce the behavior in question

This issue was seen in production (Edge devices) where rabbitmq kept on crashing with the error.

Since the env is edge devices, couldn't run diagnostics or any further debugging.

### advanced.config

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location
<details>

PASTE advanced.config HERE, BETWEEN BACKTICKS

</details>


### Application code

<details>

```python
# PASTE CODE HERE, BETWEEN BACKTICKS

Kubernetes deployment file

# Relevant parts of K8S deployment that demonstrate how RabbitMQ is deployed
# PASTE YAML HERE, BETWEEN BACKTICKS

What problem are you trying to solve?

Want to understand why the crash happened. Is it something to do with the way clients connect/ creation of queues.

Answered by michaelklishin

Nov 22, 2024

@rayate2410 had you actually read the community support policy as you claimed on the form, you'd know that RabbitMQ 3.x is out of community support.

RabbitMQ did not crash. "Crash" is a term the runtime uses for exceptions.

View full answer

michaelklishin · 2024-11-22T08:21:07Z

michaelklishin
Nov 22, 2024
Maintainer

@rayate2410 had you actually read the community support policy as you claimed on the form, you'd know that RabbitMQ 3.x is out of community support.

RabbitMQ did not crash. "Crash" is a term the runtime uses for exceptions.

0 replies

michaelklishin · 2024-11-22T08:24:15Z

michaelklishin
Nov 22, 2024
Maintainer

It is very difficult to read a log snippet that is a handful of very long lines.

There are some things that stand out immediately:

The node was told to shut down using rabbitmqctl shutdown or similar: broker forced connection closure with reason 'shutdown'
That caused an internally used library, an AMQP 0-9-1 client which STOP still uses, to stop
And as a result, a STOMP connection had to abruptly terminate, which resulted in this exception that the runtime logged

Since the node was shutting down, this is nothing more than log noise.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RCA for an exception on a RabbitMQ 3.12 node #12781

{{title}}

PASTE LOG HERE, BETWEEN BACKTICKS

PASTE LOG HERE, BETWEEN BACKTICKS

RabbitMQ configuration

PASTE advanced.config HERE, BETWEEN BACKTICKS

Replies: 2 comments

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

RCA for an exception on a RabbitMQ 3.12 node #12781

rayate2410 Nov 22, 2024

Community Support Policy

RabbitMQ version used

Erlang version used

Operating system (distribution) used

How is RabbitMQ deployed?

rabbitmq-diagnostics status output

Logs from node 1 (with sensitive values edited out)

PASTE LOG HERE, BETWEEN BACKTICKS

PASTE LOG HERE, BETWEEN BACKTICKS

RabbitMQ configuration

PASTE advanced.config HERE, BETWEEN BACKTICKS

Kubernetes deployment file

What problem are you trying to solve?

Replies: 2 comments

michaelklishin Nov 22, 2024 Maintainer

michaelklishin Nov 22, 2024 Maintainer

rayate2410
Nov 22, 2024

michaelklishin
Nov 22, 2024
Maintainer

michaelklishin
Nov 22, 2024
Maintainer