Home

Welcome to the Pi-Cluster wiki!

Documenting my experience setting up a kubernetes cluster on a set of RaspberryPi 4 model Bs (4Gb)

Setting up the Pis

I'm using Debian as the OS for this, install the OS, set a root password and create a user account. Make sure they're all accessible via ssh.

As root: apt -y update & apt -y upgrade

Then install docker.io

Update the apt package index and install packages to allow apt to use a repository over HTTPS: apt -y install apt-transport-https ca-certificates curl gnupg lsb-release

Add Docker’s official GPG key: curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

Use the following command to set up the stable repository: echo "deb [arch=arm64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null

Update the apt package index: apt update

Install docker: apt -y install docker-ce docker-ce-cli containerd.io

Confirm docker is installed correctly: docker run hello-world

Check that you get the "Hello from Docker!" message - if so docker is installed correctly!

Confirm you can see the hello world container: docker container ls -a

Remove the container to free up space: docker rm $(docker ps -a -q)

Confirm its gone with: docker container ls -a

Restart machine: systemctl reboot

Once rebooted we need to do some configuration: docker info

First, change the default cgroups driver Docker uses from cgroups to systemd to allow systemd to act as the cgroups manager and ensure there is only one cgroup manager in use. This helps with system stability and is recommended by Kubernetes. To do this, create or replace the /etc/docker/daemon.json file with:

cat > /etc/docker/daemon.json <<EOF { "exec-opts": ["native.cgroupdriver=systemd"], "log-driver": "json-file", "log-opts": { "max-size": "100m" }, "storage-driver": "overlay2" } EOF

sed -i '$ s/$/ cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1 swapaccount=1/' /boot/firmware/cmdline.txt

systemctl reboot

docker info

confirm Cgroup Driver: systemd

With these changes, Docker and the kernel should be configured as needed for Kubernetes. Reboot the Raspberry Pis, and when they come back up, check the output of docker info again. The Cgroups driver is now systemd, and the warnings are gone.

According to the documentation, Kubernetes needs iptables to be configured to see bridged network traffic. You can do this by changing the sysctl config:

cat <<EOF | tee /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 EOF

systemctl reboot

apt update && apt install -y apt-transport-https curl *nothing should be installed here

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -

cat <<EOF | tee /etc/apt/sources.list.d/kubernetes.list deb https://apt.kubernetes.io/ kubernetes-xenial main EOF

apt update

apt install kubelet kubeadm kubectl

apt-mark hold kubelet kubeadm kubectl

set the hostnames of the machines

nano /etc/hostnames

systemctl reboot

Set the static IP addresses of the nodes

nano /etc/network/interfaces.d/eth0

systemctl reboot

Initialise the control plane

TOKEN=$(kubeadm token generate) echo $TOKEN

kubeadm init --token=${TOKEN} --kubernetes-version=v1.20.4 --pod-network-cidr=10.244.0.0/16

mkdir -p $HOME/.kube cp -i /etc/kubernetes/admin.conf $HOME/.kube/config chown $(id -u):$(id -g) $HOME/.kube/config

curl -sSL https://raw.githubusercontent.com/coreos/flannel/v0.13.0/Documentation/kube-flannel.yml | kubectl apply -f -