DNS peer autodiscovery stopped working in 3.12 -> 3.13 upgrade

[bkienker]

I have been running a 3-node RabbitMQ cluster with the DNS autodiscovery cluster formation backend for a while now. I recently attempted to upgrade from 3.12.14 to 3.13.1 and found that the nodes no longer form a cluster. I attempted the upgrade in a development environment by stopping and removing all existing instances except for their DNS records, then starting one node, letting it boot, and starting the remaining two nodes.

To begin with, I did not make any modifications to the rabbitmq.conf or any port mappings to the containers running RabbitMQ. Having gone through the release notes and latest docs, I do not see where any changes to the DNS peer autodiscovery configuration would cause the feature to stop working between 3.12 and 3.13.

I am stumped because it appears I am the only one having this issue. I feel confident it is something to do with the 3.12 -> 3.13 upgrade because if I revert the RabbitMQ version to 3.12.14 with the exact same configuration, the cluster forms exactly the way it always has. I've enabled debug logging on multiple nodes, and I have not observed any errors in the DNS autodiscovery process, or really any errors at all.

Were there any changes to DNS autodiscovery in version 3.13 that would prevent a previously-working cluster from peering correctly? Thanks in advance for any advice you can offer.

[lukebakken]

Please re-try using version 3.13.7. Thank you.

[bkienker]

Thank you for the extremely quick reply. Unfortunately, I am observing the same behavior with version 3.13.7. Starting node 1 is normal, but node 2 does not discover node 1 on startup and forms its own cluster.

[lukebakken]

In your environment, if you create a FRESH 3.13.7 cluster using DNS discovery, does it succeed?

[bkienker]

Thank you, here's that information. The startup output for each RMQ node is attached as a file, for readability. Please let me know if you need anything else and I will provide it ASAP.

Here's the DNS bits:

01

service_account@rabbitmq-01 $ grep cluster_formation.dns.hostname .rabbitmq/rabbitmq.conf
cluster_formation.dns.hostname = rabbitmq-dev.subdomain.mydomain.net
service_account@rabbitmq-01 $ dig +short rabbitmq-dev.subdomain.mydomain.net
10.0.0.39
10.0.0.47
10.0.0.40
service_account@rabbitmq-01 $ dig +short -x 10.0.0.39
rabbitmq-01.mydomain.net.
service_account@rabbitmq-01 $ dig +short -x 10.0.0.40
rabbitmq-02.mydomain.net.
service_account@rabbitmq-01 $ dig +short -x 10.0.0.47
rabbitmq-03.mydomain.net.

02

service_account@rabbitmq-02 $ grep cluster_formation.dns.hostname .rabbitmq/rabbitmq.conf
cluster_formation.dns.hostname = rabbitmq-dev.subdomain.mydomain.net
service_account@rabbitmq-02 $ dig +short rabbitmq-dev.subdomain.mydomain.net
10.0.0.47
10.0.0.39
10.0.0.40
service_account@rabbitmq-02 $ dig +short -x 10.0.0.39
rabbitmq-01.mydomain.net.
service_account@rabbitmq-02 $ dig +short -x 10.0.0.40
rabbitmq-02.mydomain.net.
service_account@rabbitmq-02 $ dig +short -x 10.0.0.47
rabbitmq-03.mydomain.net.

03

service_account@rabbitmq-03 $ grep cluster_formation.dns.hostname .rabbitmq/rabbitmq.conf
cluster_formation.dns.hostname = rabbitmq-dev.subdomain.mydomain.net
service_account@rabbitmq-03 $ dig +short rabbitmq-dev.subdomain.mydomain.net
10.0.0.40
10.0.0.47
10.0.0.39
service_account@rabbitmq-03 $ dig +short -x 10.0.0.39
rabbitmq-01.mydomain.net.
service_account@rabbitmq-03 $ dig +short -x 10.0.0.40
rabbitmq-02.mydomain.net.
service_account@rabbitmq-03 $ dig +short -x 10.0.0.47
rabbitmq-03.mydomain.net.

rabbitmq-03.txt
rabbitmq-02.txt
rabbitmq-01.txt

[lukebakken]

The reverse DNS lookups on rabbitmq-03 don't appear to return the expected hostnames - is there a mistake in what you pasted, or is this an actual issue?

[bkienker]

That's a copy/paste mistake, apologies, I will fix it.

[lukebakken]

I'll see how we test this part of RabbitMQ, and may be able to find time to investigate soon-ish.

[bkienker]

Thank you for your help!

[lukebakken]

@bkienker from your 3.13 environment, can you start up RabbitMQ, and send us the logs? DNS peer discovery should log lines with Addresses discovered via in them. You can check your 3.12 environment for these log entries as well.

[michaelklishin]

Logs at debug level to be more specific.

[bkienker]

Thank you both, I have started each cluster node from a blank slate and attached all logs. I'm not seeing the Addresses discovered via line. Nothing else is really standing out to me. Please let me know what additional information I can provide to assist.

rabbitmq-03.log
rabbitmq-02.log
rabbitmq-01.log

[lukebakken]

Are you sure it's a blank state? There is no indication that peer discovery/clustering is even being attempted in those log files.

[lukebakken]

I'm getting a local env set up to test. I'm not using k8s

[bkienker]

I've attempted to get as blank a slate as possible. Attached are logs from 01 and 02, now running the official image instead of the Bitnami one, after running podman system reset. They seem to be logging much more about DNS peer discovery, but rabbitmqctl cluster_status still shows that both nodes form a cluster only containing themselves.

rmq-01.log
rmq-02.log

[lukebakken]

Yep, that is more useful. Here are the relevant log lines from rabbitmq-01:

Sep 05 12:43:42 rabbitmq-01 rabbitmq[4071336]: 2024-09-05 16:43:42.331972+00:00 [info] <0.253.0> DB: virgin node -> run peer discovery
Sep 05 12:43:42 rabbitmq-01 rabbitmq[4071336]: 2024-09-05 16:43:42.364934+00:00 [info] <0.253.0> Addresses discovered via A records of rabbitmq-dev.mydomain.net: 10.90.70.47, 10.90.70.40, 10.90.70.39
Sep 05 12:43:42 rabbitmq-01 rabbitmq[4071336]: 2024-09-05 16:43:42.368992+00:00 [info] <0.253.0> Addresses discovered via AAAA records of rabbitmq-dev.mydomain.net:
Sep 05 12:43:42 rabbitmq-01 rabbitmq[4071336]: 2024-09-05 16:43:42.369059+00:00 [debug] <0.253.0> Peer discovery: backend returned the following configuration:
Sep 05 12:43:42 rabbitmq-01 rabbitmq[4071336]: 2024-09-05 16:43:42.369059+00:00 [debug] <0.253.0>   {ok,{['rabbit@rabbitmq-03','rabbit@rabbitmq-02',rabbit@host],
Sep 05 12:43:42 rabbitmq-01 rabbitmq[4071336]: 2024-09-05 16:43:42.369059+00:00 [debug] <0.253.0>        disc}}
...
...
...
Sep 05 12:43:42 rabbitmq-01 rabbitmq[4071336]: 2024-09-05 16:43:42.629572+00:00 [info] <0.253.0> Peer discovery: discovered no peer nodes to cluster with. Some discovery backends can filter nodes out based on a readiness criteria. Enabling debug logging might help troubleshoot.

Notice that the reverse DNS lookups on those three IP addresses returned the following host names:

• rabbitmq-03
• rabbitmq-02
• host

Obviously, the last one is not correct.

On node rabbitmq-01, can you please do the following?

• A reverse DNS on the IP address (dig -x) for that node to see what is returned?
• Run the following erl command, using that node's IP (replace the X):

  erl -noinput -eval 'io:format("~tp~n",[inet:gethostbyaddr("10.90.70.X")]),init:stop().'
  

[bkienker]

That is very odd. Here's the dig from the host VM:

service_account@rabbitmq-01 $ dig +noall +answer -x 10.90.70.39
39.70.90.10.in-addr.arpa. 300	IN	PTR	rabbitmq-01.mydomain.net.

From the RabbitMQ container shell:

root@rabbitmq-01:/# erl -noinput -eval 'io:format("~tp~n",[inet:gethostbyaddr("10.90.70.39")]),init:stop().'
{ok,{hostent,"host.containers.internal",[],inet,4,[{10,90,70,39}]}}

Comparing to the same DNS query for the 02 node's IP address:

root@rabbitmq-01:/# erl -noinput -eval 'io:format("~tp~n",[inet:gethostbyaddr("10.90.70.40")]),init:stop().'
{ok,{hostent,"rabbitmq-02.mydomain.net",[],inet,4,[{10,90,70,40}]}}

I've found the culprit -- there's an entry in the container's /etc/hosts file that is superseding the DNS result:

root@rabbitmq-01:/# grep host.containers.internal /etc/hosts
10.90.70.39	host.containers.internal

After removing that line, the DNS query is returning the correct FQDN:

root@rabbitmq-01:/# erl -noinput -eval 'io:format("~tp~n",[inet:gethostbyaddr("10.90.70.39")]),init:stop().'
{ok,{hostent,"rabbitmq-01.mydomain.net",[],inet,4,[{10,90,70,39}]}}

Thank you for the assistance. I will see if I can stop podman from "helping" me with the /etc/hosts file. I apologize for the wild goose chase.

[michaelklishin]

@bkienker this sounds like something we must document.

Also note that Erlang allows you to tweak DNS resolution a great deal and likely bypass /etc/hosts entirely.

[lukebakken]

I apologize for the wild goose chase.

No worries! I'm still mystified as to how this works fine with RabbitMQ 3.12 since the DNS lookup code hasn't changed. What version of Erlang are you using in your 3.12 environments?

[michaelklishin]

Peer discovery did change in 3.13, so its place in the node boot sequence may matter. For example, with 3.12 it could have happened before Podman had a chance to mess with the /etc/hosts file.