DevOps Insights with Dennis: Kubernetes cluster orchestration with Ansible and Terraform

DevOps Insights with Dennis: Kubernets cluster orchestration with Ansible and Terraform

DevOps Insights with Dennis

Did you know about the Opslogix DevOps Upskill Program? Through the program, skillful IT consultants improve their DevOps knowledge through a combination of theoretical and practical training.

In this new blog series, our DevOps consultant Dennis will share some of the insights he has gained through the program.

Keep on reading this first part of the blog series to learn more about Dennis' insights on Kubernetes cluster orchestration with Ansible and Terraform on Proxmox VE.

Introduction

In the fast-paced world of DevOps and container orchestration, deploying a Kubernetes cluster efficiently is a crucial step toward building scalable and resilient applications. In this blog post, we'll explore a project that leverages the power of Ansible and Terraform to automate the deployment of a Kubernetes cluster on Proxmox VE. This combination of tools not only simplifies and speeds up the process but also ensures consistency and repeatability in the deployment.

The resulting GitHub-project containing Ansible and Terraform code can be found here.

Background

As a DevOps practitioner, Kubernetes is a central part of my toolbox. I have previously been using k3s as my primary home lab Kubernetes cluster, which is a lightweight version of Kubernetes that comes with some required components preinstalled, that you otherwise have to chose and install manually. Until recently the cluster worked fine, but during an upgrade I got an unexpected behavior which made me look for the control of software that standard Kubernetes provides.

Because this is a home lab environment, things get broken all the time and a quick way to restore things is essential in order to keep things going. As in production, a way to keep track of infrastructural changes is also vital in order to maintain a understanding of the environment as a whole, in order to protect it from things like malicious actors or bad configurations. While I do keep rigorous backups, I wanted a way to spin up, or down, my cluster (or several) with the push of a button in order to reset it - i.e. automation! That's why I looked to the tools Ansible and Terraform solve my problem. 

Why Ansible and Terraform?

Ansible and Terraform are two popular open-source tools known for their infrastructure-as-code (IaC) capabilities. Ansible excels in configuration management, allowing for seamless automation of tasks such as package installation, configuration file management, and service orchestration. Terraform, on the other hand, specializes in provisioning and managing infrastructure resources across various cloud providers and virtualization platforms.

By combining Ansible and Terraform, developers and operations teams can create a powerful synergy that addresses both configuration and provisioning aspects of infrastructure management.

Setting the Stage: Proxmox Hypervisor

Proxmox Virtual Environment (Proxmox VE) is an open-source virtualization platform that combines two virtualization technologies: KVM (Kernel-based Virtual Machine) for virtual machines and LXC (Linux Containers) for lightweight container-based virtualization. Proxmox provides a user-friendly web interface and powerful command-line tools for managing virtualized environments.

For this project, I am using Opslogix's test environment consisting of 3 Proxmox nodes that are ready to go. If you want to get started with Proxmox, take a look here.

The Project Workflow

1. Infrastructure Provisioning with Terraform

Terraform scripts are used to define the infrastructure requirements for the Kubernetes cluster. This includes virtual machines, networks, storage, and any other resources necessary for the cluster. The Terraform configuration files, written in HashiCorp Configuration Language (HCL), provide a declarative syntax for specifying the desired state of the infrastructure. 

In the below code snippet from the main Terraform file, we declare how our Virtual Machines should be provisioned - e.g. how much resources to give, the host name, IP addresses, and which template to use. 

resource "proxmox_vm_qemu" "proxmox_kubernetes_vms" {
  count = var.x_number_of
  name  = "${var.vm_name}-${count.index + 1}"
  ipconfig0   = "gw=${var.ip_gateway},ip=${var.ip_addresses[count.index]}"
  target_node = "proxmox_hostname_or_ip"
  onboot      = true
  # Same CPU as the Physical host, possible to add cpu flags
  # Ex: "host,flags=+md-clear;+pcid;+spec-ctrl;+ssbd;+pdpe1gb"
  cpu   = "host"
  numa  = false
  clone = "${var.template_vm_name}"
  full_clone = true
  os_type    = "cloud-init"
  agent      = 1
  ciuser     = var.user
  memory     = var.memory
...

2. Terraform deployment with Ansible

Once the infrastructure is declared in Terraform, Ansible takes over for actual deployment. Ansible playbooks use YAML (Yet Another Markup Language) to define tasks to be performed and in which sequence.

In the snippet below from the Ansible host provisioning file, we can see how Ansible will create the Terraform plan and then apply it in sequence.

- name: Creating k8s VMs Terraform plan
  community.general.terraform:
    project_path: ""
    state: planned
    plan_file: "/k8s_hosts.tfplan"
    force_init: true

- name: Confirm terraform plan is present
  ansible.builtin.wait_for:
    path: "/k8s_hosts.tfplan"
    state: present

- name: Apply terraform plan
  community.general.terraform:
    project_path: ""
    plan_file: "/k8s_hosts.tfplan"
    state: present
  register: outputs

3. Kubernetes Deployment

With the infrastructure provisioned and nodes configured, the final step is deploying Kubernetes. Ansible playbooks can be used to install Kubernetes components, initialize the cluster, and join nodes to the cluster.

In the snippet below, the Ansible initialize cluster file creates the cluster and saves the join command, which is printed at the end of the cluster creation, to use for joining the worker nodes to the cluster.

- name: Initialize cluster
  ansible.builtin.command: "kubeadm init --pod-network-cidr="
  register: result
  changed_when: "result.rc == 0"
  when: "reset_cluster is succeeded"


- name: Run script to add kubeconfig
  ansible.builtin.script: add_kubeconfig.sh


- name: Get the token for joining the worker nodes
  ansible.builtin.command: kubeadm token create --print-join-command
  register: kubernetes_join_command
  changed_when: false
  failed_when: "kubernetes_join_command.rc != 0"

Benefits of Automation

1. Consistency: Automation ensures that each deployment follows the same set of steps, eliminating variations and reducing the risk of configuration drift.
2. Repeatability: With IaC tools like Ansible and Terraform, the entire infrastructure can be recreated at any time, providing a reliable and reproducible deployment process.
3. Time Efficiency: Manual tasks are prone to errors and can be time-consuming. Automation significantly speeds up the deployment process, allowing teams to focus on more strategic aspects of their projects.
4. Scalability: As your infrastructure needs grow, automation allows for easy scaling by adjusting parameters in the configuration files without the need for manual intervention.

Conclusion

In conclusion, the project showcases the power of combining Ansible and Terraform to automate the deployment of a Kubernetes cluster on a Proxmox hypervisor. This approach not only streamlines the deployment process but also enhances the reliability and scalability of the infrastructure. By adopting infrastructure as code practices, development, and operations teams can achieve greater efficiency and agility in managing complex systems.

Do you want to learn more about the Opslogix DevOps Upskill Program?