Crucible Production Kubernetes

Terraform, Ansible and kubernetes code to install a production instance of Crucible

This stack utilizes Rancher k3s distrobution with a rancher front end for web role based access to the cluster.

The kubernetes cluster utilizes the kubernetes ingress nginx controller, with a longhorn persistant storage backend and MetalLB for IP allocation.

Requirements

• vSphere Server with Distributed Port Groups.
• Ubuntu 20.04 template, with snapshot.
• Ubuntu 20.04 Desktop Jump Box.

Ubuntu Jump Box

1. Create an Ubuntu 20.04 Desktop VM as a jump box.

   • 30GB+ HD
   • 4 processors
   • 6GB RAM
   • Install Ubuntu Desktop, git and vscode.

2. Clone this repository: git clone https://github.com/sei-noconnor/k3s-production.git

3. Navigate to: cd k3s-production.

4. Rename env.example to env and set the variables in this file, many required defaults are intentionally missing, pay close attention to TOKEN and PASS values.

5. Run sudo ./prep. This will install the binaries needed for the appliance, including terraform, ansible and the kubernetes binaries

ENV File

The env file is a set a variables for customizing the stack to your unique infrastructure. It is laid out like an ini file and similar in function to docker-compose env file. The ./setup script will replace these values in the relevant files.

Networking

A standard network will need to be avaialble and a block of apporimatly 10 IPs is required 7 for kubernetes, 3 for various ingress adresses.

Set the variables in the [network] section of the env file.

DNS

This stack uses an kubernetes ingress nginx. Hostnames must be used when accessing the applications. While most Crucible applications use path based routing to limit entires in DNS there are third party applications that need entires as well. If you are setting up an initial Proof of Concept a host file entry can be used.

Deploying Kubernetes VMs

This repo contains terraform to deploy VMs needed for kubernetes. it utilizes your existing ubuntu template. You may wish to modify the VMs configuration such as preocessors and RAM, or last octect of IP. edit the terraform/variables.auto.tfvars at the bottom you can modify the vm configuration.

1. within the terraform directory run terraform init
2. run terraform plan and verify that the correct resources will be created.
3. run terraform apply again verify the resources and confirm you want to apply.
4. your VMs will begin to be provisioned and will be accessable at their specified IPs listed in the output of the terraform.

Installing Kubernetes with Ansible.

This repo contains an ansible playbook to install k3s master, slave and worker nodes. modify k3s-ansible/inventory/sample/hosts.ini with the correct values based on the terraform output.

Certificate

Certificates can be generated by setting the GENERATE_CERTS value in the env file to true. you will still need to provide your vcenter cert.

If you are providing your own certificate be sure to place certificates in common/certs with the following file names.

Type	Filename
host	host.pem
host key	host-key.pem
root CA	root-ca.pem
vSphere	vsphere.pem

URLs

---

Replace crucible.io with the <DOMAIN> environment URL if changed.

Application	URL
Alloy	crucible.io/alloy
Alloy API	crucible.io/alloy/api
Caster	crucible.io
Caster API	crucible.io/api
Identity	crucible.io
Identity API	crucible.io/api
Player	crucible.io
Player API	crucible.io/api
Steamfitter	crucible.io
Steamfitter API	crucible.io/api
VM	cucible.io
VM API	crucible.io/api
VM Console	crucible.io

3rd Party URLs

---

3rd Party Applications	URL
GitLab	gitlab.crucible.io
StackStorm	stackstorm.crucible.io