Webhook stopped working. How to debug?

[Stolz]

I have been successfully using the vault-secrets-webhook in several of my clusters for around a year. However a few days ago the webhook stopped working in one of the clusters: the pods do not contain a copy-vault-env init container and the environment variables contain secret placeholders instead of the actual secrets.

In the past I had a similar issues but it was intermittent and always got solved by restarting deployment/vault-secrets-webhook. However the issue now seems permanent and restarting deployment/vault-secrets-webhook or deployment/vault-operator does not help. If I deploy the manifests shown below it works fine in all of my clusters but one, which confirm the manifest is correct but something is wrong with that only cluster. Since neither deployment/vault-secrets-webhook, deployment/vault-operator or any other resources listed below show any error in the logs... how can I debug what's going on?

---
apiVersion: v1
kind: Namespace
metadata:
  name: deleteme
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: deleteme
    function: debug
  name: deleteme
  namespace: deleteme
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: deleteme
    function: debug
  name: deleteme
  namespace: deleteme
spec:
  replicas: 1
  selector:
    matchLabels:
      app: deleteme
      function: debug
  template:
    metadata:
      annotations:
        vault.security.banzaicloud.io/vault-addr: http://vault.management.svc.cluster.local:8200
        vault.security.banzaicloud.io/vault-path: kubernetes
        vault.security.banzaicloud.io/vault-role: deleteme
        vault.security.banzaicloud.io/vault-skip-verify: "true"
      labels:
        app: deleteme
        function: debug
    spec:
      containers:
        - env:
            - name: WHOAMI_NAME
              value: vault:secret/data/deleteme#PASSWORD
            - name: WHOAMI_PORT_NUMBER
              value: "8080"
          image: traefik/whoami
          name: deleteme
          ports:
            - containerPort: 8080
          resources:
            limits:
              cpu: 100m
              memory: 128M
            requests:
              cpu: 10m
              memory: 64M
      hostname: deleteme
      serviceAccountName: deleteme # Required to retrieve secrets from Vault
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: deleteme
    function: debug
  name: deleteme
  namespace: deleteme
spec:
  ports:
    - name: http
      port: 80
      targetPort: 8080
  selector:
    app: deleteme
    function: debug
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  labels:
    app: deleteme
    function: debug
  name: deleteme
  namespace: deleteme
spec:
  ingressClassName: nginx
  rules:
    - host: deleteme.example.net
      http:
        paths:
          - backend:
              service:
                name: deleteme
                port:
                  name: http
            path: /
            pathType: Prefix

The Pod is created but it has no init contianer

kubectl get pod -o jsonpath="{.spec['containers','initContainers'][*].name}" deleteme-657fbcb679-n9x5p
deleteme

And obviously the secrets are not injected

curl http://deleteme.example.net

Name: vault:secret/data/deleteme#PASSWORD
Hostname: deleteme
IP: 127.0.0.1
IP: 10.124.240.185
IP: fe80::3cc0:9bff:fe49:7129
RemoteAddr: 10.124.240.165:56126
GET / HTTP/1.1
Host: deleteme.example.net
User-Agent: curl/8.1.2
Accept: */*
X-Forwarded-For: 10.124.240.4
X-Forwarded-Host: deleteme.example.net
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Scheme: http
X-Real-Ip: 10.124.240.4
X-Scheme: http

[akijakya]

Hi @Stolz, first of all, thanks for using Bank-Vaults! A couple of questions:
Do you use the same version of the operator, webhook etc. on all of your clusters?
On the cluster where the webhook doesn't seem to work, do the webhook logs show the Admission review request handled info log for the pod it fails to inject into?
Is this problem present with all of the deployed workloads or just one, or ones deployed to a specific namespace?
What happens if you set podsFailurePolicy: Fail in the webhook's config?

[Stolz]

Hi @akijakya,

Thanks a lot for your help. In order to answer your last questions I have redeployed the chart to set podsFailurePolicy=Fail and that has solved the problem. I'm not sure if redeploying the chart did the trick or it was setting podsFailurePolicy=Fail .

Since my Helm release was using the charts from the now archived Helm repo I did not try to redeploy it, I just restart it. This command allowed me to change the values of the release without upgarding the chart version

» helm upgrade vault-secrets-webhook banzaicloud-stable/vault-secrets-webhook --version 1.16.0 --reuse-values --set podsFailurePolicy=Fail

Now everything works as expected. Just in case it hels anyone having the same issue in the future that may end up here, here are the answers to your other questions.

Do you use the same version of the operator, webhook etc. on all of your clusters?

See below (the one failing is the first one, the other two work fine). As you can see, the version of the helm chart is slightly different but the operator application version macthes in all of the clusters and the webhook application version macthes in the first and second cluster.

» kubectl config use-context dev && helm list
NAME                    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                           APP VERSION
vault                   management      7               2022-11-16 10:00:00.600666 +0800 HKT    deployed        vault-0.22.0                    1.11.3
vault-operator          management      8               2022-11-16 10:02:18.549307 +0800 HKT    deployed        vault-operator-1.16.1           1.16.0
vault-secrets-webhook   management      5               2022-11-16 10:04:46.261233 +0800 HKT    deployed        vault-secrets-webhook-1.16.0    1.16.0

» kubectl config use-context uat && helm list
NAME                    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                           APP VERSION
vault                   management      2               2023-03-14 14:00:26.118139 +0800 HKT    deployed        vault-0.23.0                    1.12.1
vault-operator          management      2               2023-06-01 17:35:55.181097 +0800 HKT    deployed        vault-operator-1.16.0           1.16.0
vault-secrets-webhook   management      2               2023-06-01 18:06:17.759092 +0800 HKT    deployed        vault-secrets-webhook-1.16.0    1.16.0

» kubectl config use-context prod && helm list
NAME                    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                           APP VERSION
vault                   management      1               2022-09-14 18:06:43.38295 +0800 HKT     deployed        vault-0.21.0                    1.11.2
vault-operator          management      1               2022-09-15 14:26:22.654932 +0800 HKT    deployed        vault-operator-1.16.0           1.16.0
vault-secrets-webhook   management      4               2023-09-20 12:44:50.016399 +0800 HKT    deployed        vault-secrets-webhook-1.19.0    1.19.0

Is this problem present with all of the deployed workloads or just one, or ones deployed to a specific namespace?

The problem is present with all workloads and all namespaces I have tried

On the cluster where the webhook doesn't seem to work, do the webhook logs show the Admission review request handled info log for the pod it fails to inject into?

If I deploy the manifest mentioned in my previous message I see these logs ...

» kubectl apply -f deleteme-deployment.yml
Creating/Updating K8s resources ...
namespace/deleteme created
serviceaccount/deleteme created
deployment.apps/deleteme created
service/deleteme created
ingress.networking.k8s.io/deleteme created

» kubectl get pods -o wide
NAME                        READY   STATUS    RESTARTS   AGE     IP
deleteme-657fbcb679-9594x   1/1     Running   0          10s     10.124.240.189

» kubectl logs deploy/deleteme
2023/11/29 05:56:16 Starting up on port 8080

» kubectl logs -n vault deploy/vault-secrets-webhook # This deployment has only one replica
2023/11/29 05:56:59 http: TLS handshake error from 10.124.240.74:55788: EOF
2023/11/29 05:56:59 http: TLS handshake error from 10.124.240.74:55800: remote error: tls: bad certificate
2023/11/29 05:56:59 http: TLS handshake error from 10.124.240.129:39832: remote error: tls: bad certificate
2023/11/29 05:57:00 http: TLS handshake error from 10.124.240.129:39834: EOF

As you can see, there is no mention of the pod I just created (x.x.x.189) but it mentions two other IPs (x.x.x.74 and x.x.x.189) that belong to the konnectivity-agent pods

» kubectl get --all-namespaces --output json pods | jq '.items[] | select(.status.podIP=="10.124.240.74")' | jq .metadata.namespace,.metadata.name
"kube-system"
"konnectivity-agent-68cdcd54c6-45j2q"

» kubectl get --all-namespaces --output json pods | jq '.items[] | select(.status.podIP=="10.124.240.129")' | jq .metadata.namespace,.metadata.name
"kube-system"
"konnectivity-agent-68cdcd54c6-z26p4"

Since the timestamps don't exactly match and the messages keep repeating even when the original pod is deleted I assume they are not related with my issue.

[akijakya]

I'm happy @Stolz that your issue got resolved, and thanks for your time in providing these details!