Kubernetes with Bitnami: rabbitmq-13.12.12 HTTP API-based readiness probe fails

[pritam-d29]

Hi,

Seek support in one of the issues, that I am facing with Rabbitmq version-13.12.12.
The rabbitmq here I am using with my Kubernetes-1.28.8 in IPv6 env.

I am deploying the rabbitmq pod with helm based cli.
Pods is not coming up. Please find the details for your reference -

Upon Describing

Name:             rabbitmq-0
Namespace:        rabbitmq
Priority:         0
Service Account:  rabbitmq
Node:             kubeworker7.cluster.local/2001:db8::7
Start Time:       Fri, 12 Jul 2024 12:58:05 +0530
Labels:           app.kubernetes.io/instance=rabbitmq
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=rabbitmq
                  app.kubernetes.io/version=3.12.12
                  apps.kubernetes.io/pod-index=0
                  controller-revision-hash=rabbitmq-c4665fcd6
                  helm.sh/chart=rabbitmq-12.10.0
                  statefulset.kubernetes.io/pod-name=rabbitmq-0
Annotations:      checksum/config: c91b14bfecbe21ce5da4c5d3c7af53979be317a1d87066d9f53338aad2abd290
                  checksum/secret: 2689036789a39be6114657fb10e21a522b160b89a7405612495014f512972e58
                  cni.projectcalico.org/containerID: 5664863ce77d9bca588cc1183d899ba616e18f7b14b5eeb6f6d0418d7feb9792
                  cni.projectcalico.org/podIP: 3ffe:10::7cf4:601c:a077:73d2/128
                  cni.projectcalico.org/podIPs: 3ffe:10::7cf4:601c:a077:73d2/128
Status:           Running
IP:               3ffe:10::7cf4:601c:a077:73d2
IPs:
  IP:           3ffe:10::7cf4:601c:a077:73d2
Controlled By:  StatefulSet/rabbitmq
Init Containers:
  ipv6-init:
    Container ID:  containerd://a8f95e921af09e6489a486e90d802996436bcc6094bb014eff8d9c5168633de2
    Image:         docker.io/busybox:1.33.1
    Image ID:      docker.io/library/busybox@sha256:f7ca5a32c10d51aeda3b4d01c61c6061f497893d7f6628b92f822f7117182a57
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
      echo "{inet6, true}." > /ipv6/erl_inetrc
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 12 Jul 2024 12:58:06 +0530
      Finished:     Fri, 12 Jul 2024 12:58:06 +0530
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /ipv6 from ipv6-cfg (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-65n7d (ro)
Containers:
  rabbitmq:
    Container ID:    containerd://ffc535782e2c7611e32c6d4ccebf3cf3163a61dce821f4c619bfd28fedd38236
    Image:           docker.io/bitnami/rabbitmq:3.12.13-debian-12-r2
    Image ID:        docker.io/bitnami/rabbitmq@sha256:34e64ea7306296d93f530adaf722ced6f150f69596ce75444b46bd5c5104fa0b
    Ports:           5672/TCP, 25672/TCP, 15672/TCP, 4369/TCP, 5671/TCP
    Host Ports:      0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
    SeccompProfile:  RuntimeDefault
    State:           Waiting
      Reason:        CrashLoopBackOff
    Last State:      Terminated
      Reason:        Error
      Exit Code:     1
      Started:       Fri, 12 Jul 2024 13:30:56 +0530
      Finished:      Fri, 12 Jul 2024 13:33:04 +0530
    Ready:           False
    Restart Count:   8
    Liveness:        exec [sh -ec curl -f --user test:$RABBITMQ_PASSWORD 127.0.0.1:15672/api/health/checks/virtual-hosts] delay=120s timeout=50s period=30s #success=1 #failure=6
    Readiness:       exec [sh -ec curl -f --user test:$RABBITMQ_PASSWORD 127.0.0.1:15672/api/health/checks/local-alarms] delay=20s timeout=70s period=30s #success=1 #failure=3
    Environment:
      BITNAMI_DEBUG:                        false
      MY_POD_IP:                             (v1:status.podIP)
      MY_POD_NAME:                          rabbitmq-0 (v1:metadata.name)
      MY_POD_NAMESPACE:                     rabbitmq (v1:metadata.namespace)
      RABBITMQ_FORCE_BOOT:                  no
      RABBITMQ_NODE_NAME:                   rabbit@$(MY_POD_NAME).rabbitmq-headless.$(MY_POD_NAMESPACE).svc.cluster.local
      RABBITMQ_MNESIA_DIR:                  /opt/bitnami/rabbitmq/var/lib/rabbitmq/$(RABBITMQ_NODE_NAME)
      RABBITMQ_LDAP_ENABLE:                 no
      RABBITMQ_LOGS:                        -
      RABBITMQ_ULIMIT_NOFILES:              65536
      RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS:  +S 2:1
      RABBITMQ_USE_LONGNAME:                true
      RABBITMQ_ERL_COOKIE:                  <set to the key 'rabbitmq-erlang-cookie' in secret 'rabbitmq'>  Optional: false
      RABBITMQ_LOAD_DEFINITIONS:            no
      RABBITMQ_DEFINITIONS_FILE:            /app/load_definition.json
      RABBITMQ_SECURE_PASSWORD:             yes
      RABBITMQ_USERNAME:                    test
      RABBITMQ_PASSWORD:                    <set to the key 'rabbitmq-password' in secret 'rabbitmq'>  Optional: false
      RABBITMQ_PLUGINS:                     rabbitmq_management, rabbitmq_peer_discovery_k8s, rabbitmq_event_exchange, rabbitmq_auth_backend_ldap
      RABBITMQ_CTL_ERL_ARGS:                -proto_dist inet6_tcp
    Mounts:
      /bitnami/rabbitmq/conf from configuration (rw)
      /ipv6 from ipv6-cfg (rw)
      /opt/bitnami/rabbitmq/certs from certs (rw)
      /opt/bitnami/rabbitmq/var/lib/rabbitmq from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-65n7d (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-rabbitmq-0
    ReadOnly:   false
  certs:
    Type:                Projected (a volume that contains injected data from multiple sources)
    SecretName:          rabbitmq-certificates
    SecretOptionalName:  <nil>
  configuration:
    Type:                Projected (a volume that contains injected data from multiple sources)
    SecretName:          rabbitmq-config
    SecretOptionalName:  <nil>
  ipv6-cfg:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-65n7d:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  37m                 default-scheduler  Successfully assigned rabbitmq/rabbitmq-0 to kubeworker7.cluster.local
  Normal   Pulled     44m                 kubelet            Container image "docker.io/busybox:1.33.1" already present on machine
  Normal   Created    44m                 kubelet            Created container ipv6-init
  Normal   Started    44m                 kubelet            Started container ipv6-init
  Normal   Pulled     40m (x3 over 44m)   kubelet            Container image "docker.io/bitnami/rabbitmq:3.12.13-debian-12-r2" already present on machine
  Normal   Created    40m (x3 over 44m)   kubelet            Created container rabbitmq
  Normal   Started    40m (x3 over 44m)   kubelet            Started container rabbitmq
  Warning  Unhealthy  29m (x30 over 44m)  kubelet            Readiness probe failed:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 15672 after 0 ms: Couldn't connect to server
  Warning  BackOff  9m25s (x79 over 40m)  kubelet  Back-off restarting failed container rabbitmq in pod rabbitmq-0_rabbitmq(8db57004-d935-4f09-b44e-a754c6c1980b)

A snippet of output -

service output -

The exported yaml enclosed.
rab_afterChnages.log

Let me know, incase you need any further details.

Thanks

[michaelklishin]

RabbitMQ 3.12 is out of community support, although I'm certain that the issue is very unlikely to be version-specific and the pod simply fails to boot.

Since you use an older version with Bitnami, my first intuition is this classic deployment deadlock problem on Kubernetes, which is mentioned in this Kubernetes-specific part of the documentation. Here's a Stack Overflow version of the explanation with a truly awful piece of advise for the most popular answer 🤦

Besides that, start with inspecting RabbitMQ logs from all nodes. That's the most useful source of information for the problem at hand and you are ignoring it. Kubernetes logs will tell you nothing about why a node cannot finish booting.

[lukebakken]

@pritam-d29 stated in an email to rabbitmq-core (which is NOT a support email):

Unfortunately, this isn’t a first time for me. Many of my concerns have either been ignored or dismissed. I've raised issues in the community forums for Calico and Redis too, but haven’t received any support. Everyone’s insisting on a contractual agreement to move forward.

Yes, it is because the people who give away this software have to make money, somehow.

In this specific case, @michaelklishin has already given you suggestions for what to look at. LOGS!