Kubernetes IAC

Terraform Kubernetes Infrastructure as Code (IAC)

This repository was created by following the instructions in the article linked below, with modifications to suit my specific cluster configuration. Note that the datasets or datastores on my Proxmox setup may differ from yours, so adjust accordingly. My Proxmox cluster consists of three nodes and uses Ceph for efficient virtual machine management across all nodes.

Article: Talos Cluster on Proxmox with Terraform by Olav

---

Project Structure & Automation

Overview

This repository provides Infrastructure as Code (IaC) for deploying and managing a Kubernetes cluster on Proxmox using Talos and Terraform. It is designed for repeatable, automated, and declarative cluster management.

Key Features

• Declarative VM Provisioning: Proxmox VMs for control plane and worker nodes are managed via Terraform.
• Talos OS & Kubernetes Versioning: Talos and Kubernetes versions are parameterized in variables.tf for easy upgrades.
• Automated Cluster Configuration: Talos machine configurations are generated and applied automatically to each node.
• Rolling Upgrades: Change a version variable and apply to safely upgrade Talos and/or Kubernetes across your cluster.
• CI/CD Linting: A GitHub Actions workflow automatically checks Terraform formatting and lints code on pull requests and pushes to main.

How to Upgrade Talos or Kubernetes

1. Edit the version variables in variables.tf:

   variable "talos_version" {
     default = "v1.9.5"
   }
   variable "kubernetes_version" {
     default = "1.32.0"
   }
2. Run:

   terraform apply
   This triggers a rolling upgrade of your cluster nodes using the new versions.

Directory Structure

.
├── cluster.tf         # Talos cluster and machine configuration resources
├── files.tf           # Talos image download and local variables
├── providers.tf       # Terraform provider configuration
├── variables.tf       # All input variables, including versioning
├── virtual_machines.tf# Proxmox VM definitions for control plane and workers
├── .github/workflows/terraform-lint.yml # CI workflow for linting
└── README.md          # Project documentation

Automation

• Terraform Linting:
  On every PR or push to main, the .github/workflows/terraform-lint.yml workflow runs:
  • terraform fmt -check -recursive
  • tflint --recursive to ensure code quality and consistency.

---

Additional Steps

After setting up the cluster, you may find the following steps helpful.

Connect to the Talos Kubernetes Cluster

To connect to your Talos Kubernetes cluster using Terraform outputs, configure your local environment as follows:

1. Save the kubeconfig and talosconfig outputs to files on your local machine:

   terraform output -raw kubeconfig > ~/.kube/config
   terraform output -raw talosconfig > ~/.talos/config

2. Set appropriate file permissions to avoid security issues:

   chmod 600 ~/.kube/config ~/.talos/config

Set Up kubectl

To interact with the Kubernetes cluster, use kubectl. For example, to list the nodes:

kubectl get nodes

Sample output:

NAME              STATUS   ROLES           AGE   VERSION
talos-cp-01       Ready    control-plane   83s   v1.32.0
talos-cp-02       Ready    control-plane   86s   v1.32.0
talos-cp-03       Ready    control-plane   85s   v1.32.0
talos-worker-01   Ready    <none>          88s   v1.32.0
talos-worker-02   Ready    <none>          86s   v1.32.0
talos-worker-03   Ready    <none>          90s   v1.32.0

Use talosctl

• View the Dashboard:

  talosctl dashboard -n talos-cp-01

• Check Cluster Health:

  talosctl -n talos-cp-01 health

  Sample output:

  discovered nodes: ["10.0.0.73" "10.0.0.74" "10.0.0.75" "10.0.0.70" "10.0.0.71" "10.0.0.72"]
  waiting for etcd to be healthy: OK
  waiting for all k8s nodes to report ready: OK
  waiting for all control plane components to be ready: OK
  ...
  

• Health Dashboard Example:

Reset the Cluster

If you need to start over, you can taint resources and reapply the Terraform configuration:

terraform state list | xargs -n1 terraform taint
terraform apply

---

Adjust paths and configurations as needed for your environment.