Bug: tf-runner pod stuck in microk8s

[ajaydevtron]

Setup details:
Kubernetes: Micok8s 1.31
Flux: Installed with kubectl (https://fluxcd.io/flux/installation/#dev-install)
tf-controller: v0.16.0-rc.5 installed with helm chart
https://flux-iac.github.io/tofu-controller/getting_started/

Gone through a similar issue #1181 but didn’t find any way to solve for microk8s.

Git Repo

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: aws-sg-git-repo
  namespace: flux-system
spec:
  interval: 30s
  ref:
    branch: main
  timeout: 60s
  url: https://github.com/ajaydevtron/helloworld

Terraform object

apiVersion: infra.contrib.fluxcd.io/v1alpha2
  kind: Terraform
  metadata:
    name: aws-sg-tf
    namespace: flux-system
  spec:
    approvePlan: auto
    interval: 1m
    path: ./
    runnerPodTemplate:
      spec:
        envFrom:
          - secretRef:
              name: tf-aws-creds
    sourceRef:
      kind: GitRepository
      name: aws-sg-git-repo
      namespace: flux-system
    storeReadablePlan: json

terraform object event

status:
  conditions:
    - lastTransitionTime: 2025-02-24T04:51:55Z
      message: Reconciliation in progress
      reason: Progressing
      status: Unknown
      type: Ready
  observedGeneration: 1

Output from the runner:

[Feb 24 2025 10:30:57 GMT+0530] aws-sg-tf-tf-runner: 2025/02/24 05:00:57 Starting the runner... version sha

Logs from the source-controller

[Feb 24 2025 10:38:16 GMT+0530] source-controller-fc7bc69c-2ss2t: {"level":"info","ts":"2025-02-24T05:08:16.612Z","msg":"no changes since last reconcilation: observed revision 'main@sha1:710421ae5d598689ef2ec06d69d01ac08af92f45'","controller":"gitrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"GitRepository","GitRepository":{"name":"aws-sg-git-repo","namespace":"flux-system"},"namespace":"flux-system","name":"aws-sg-git-repo","reconcileID":"e07ca108-038d-4bbb-9f75-c59e2eefb1c9"} [Feb 24 2025 10:38:25 GMT+0530] source-controller-fc7bc69c-2ss2t: {"level":"info","ts":"2025-02-24T05:08:25.731Z","msg":"no changes since last reconcilation: observed revision 'main@sha1:97e5d80ca7ece8550bc71afb51042b6f9ea8d2e8'","controller":"gitrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"GitRepository","GitRepository":{"name":"s3-creation","namespace":"flux-system"},"namespace":"flux-system","name":"s3-creation","reconcileID":"642e581e-6753-4a47-b5a4-b765158bb5e0"} [Feb 24 2025 10:38:46 GMT+0530] source-controller-fc7bc69c-2ss2t: {"level":"info","ts":"2025-02-24T05:08:46.437Z","msg":"no changes since last reconcilation: observed revision 'main@sha1:710421ae5d598689ef2ec06d69d01ac08af92f45'","controller":"gitrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"GitRepository","GitRepository":{"name":"aws-sg-git-repo","namespace":"flux-system"},"namespace":"flux-system","name":"aws-sg-git-repo","reconcileID":"5b727668-4af2-4081-ac4d-15b133d2b5f3"}

Logs from the tf-controller

[Feb 24 2025 10:24:35 GMT+0530] preview-tofu-controller-utils-56bccc6fcf-6bk6b: {"level":"info","ts":"2025-02-24T04:54:35.782Z","msg":"show runner pod state: ","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"aws-sg-tf","namespace":"flux-system"},"namespace":"flux-system","name":"aws-sg-tf","reconcileID":"a3889c11-04db-45da-aad1-0112cac33626","reconciliation-loop-id":"b1a9f7d6-1db7-43a1-93fc-49a5d63ae662","start-time":"2025-02-24T04:54:35.773Z","name":"aws-sg-tf","state":"running"} [Feb 24 2025 10:25:05 GMT+0530] preview-tofu-controller-utils-56bccc6fcf-6bk6b: {"level":"error","ts":"2025-02-24T04:55:05.784Z","msg":"Hit an error","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"aws-sg-tf","namespace":"flux-system"},"namespace":"flux-system","name":"aws-sg-tf","reconcileID":"a3889c11-04db-45da-aad1-0112cac33626","reconciliation-loop-id":"b1a9f7d6-1db7-43a1-93fc-49a5d63ae662","start-time":"2025-02-24T04:54:35.773Z","function":"TerraformReconciler.lookupOrCreateRunner_000","error":"context deadline exceeded"} [Feb 24 2025 10:25:05 GMT+0530] preview-tofu-controller-utils-56bccc6fcf-6bk6b: {"level":"error","ts":"2025-02-24T04:55:05.784Z","msg":"unable to lookup or create runner","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"aws-sg-tf","namespace":"flux-system"},"namespace":"flux-system","name":"aws-sg-tf","reconcileID":"a3889c11-04db-45da-aad1-0112cac33626","reconciliation-loop-id":"b1a9f7d6-1db7-43a1-93fc-49a5d63ae662","start-time":"2025-02-24T04:54:35.773Z","error":"context deadline exceeded"} [Feb 24 2025 10:25:05 GMT+0530] preview-tofu-controller-utils-56bccc6fcf-6bk6b: {"level":"error","ts":"2025-02-24T04:55:05.785Z","msg":"Reconciler error","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"aws-sg-tf","namespace":"flux-system"},"namespace":"flux-system","name":"aws-sg-tf","reconcileID":"a3889c11-04db-45da-aad1-0112cac33626","error":"context deadline exceeded"}

[ajaydevtron]

@chanwit Could you please help here ?