Creating Kubernetes Cluster on Azure

If you want to deploy Nevis onto a Cloud infrastructure, we recommend using the Kubernetes-based installation. This installation is based on the Docker containerization and Kubernetes orchestration technologies. For a general overview of the Nevis-on-Kubernetes deployment solution, see Main Concepts.

In this tutorial, you will set up Kubernetes cluster from scratch on Azure with the needed dependencies.

For an easier installation process check "Azure Deployment Automation", where the cluster is set up automatically with nevisAdmin4 already running in a few minutes.

The tutorial creates the following resources in Azure:

• A Terraform storage account, which will hold our cluster configuration.
• A Kubernetes cluster.
• An Azure database server for MariaDB, on which we will create our databases containing the Nevis-related data.
• A container registry, which will hold our docker images.
• A virtual network, which the cluster will use.
• Two IP address, one to access the cluster, and one to the cluster itself.

See the chapter Recommendations and Limitations, which includes limitations specific to Kubernetes.

Prerequisites

This documentation uses Terraform to manage the infrastructure on the cloud service provider. Terraform allows managing the infrastructure as code, which provides additional benefits such as version control.

This tutorial provides a way to set up a Kubernetes cluster on Azure. We tested the instructions on RHEL 7.

A couple of things are required to get started with Nevis on Kubernetes:

• Have an Azure subscription and have enough permissions to create resource groups and resources. This includes Application Administrator for creating the service principal, and Owner to assign the required Role to it.
• The supported Kubernetes versions for this guide are 1.24, 1.25, 1.26, 1.27 and 1.28.
• A Linux environment with the following software pre-installed:
  • terraform: Terraform command line tool.
  • az: Azure command line interface.

This guide requires basic knowledge of Linux, Terraform and the Azure CLI. In case you are new to these topics, we recommend to use the Azure deployment automation instead.

Steps to Perform

Perform the following steps for a Kubernetes-based installation of nevisAdmin 4:

1. [Downloading Template Files]
2. [Setting Terraform Variables]
3. [Creating the Kubernetes Cluster with Terraform]

The following chapters describe each step in detail.

Downloading Template Files

All files required to set up the Kubernetes cluster are provided in the following Zip file: azure-kubernetes-cluster.zip

First, download and unzip the files.

File	Description
bootstrap/terraform-storage.tf	Defines the storage account used to store the state of the Terraform managed infrastructure.
aks-cluster.tf	Defines the actual Kubernetes cluster.
container-registry.tf	Defines a Docker container registry accessible in the cluster.
azure.tf	Links the Azure cluster/registry to a storage account.
db.tf	Defines the Azure MariaDB server.
db_config.tf	Configures the Azure MariaDb server.
variables.tf	Contains the variables used in the template files
terraform.tfvars	Contains the variable values

Setting Terraform Variables

First, fill out the missing values in the provided terraform.tfvars file.

Creating the Kubernetes Cluster with Terraform

The next step is to create a Kubernetes cluster with Terraform. First bootstrap Terraform before you can create the cluster.

Bootstrapping Terraform

Terraform can be used to easily set up a Kubernetes cluster and a Docker registry on Azure. Perform the next steps/execute the following commands:

Set the following environment variables, use the same values that were used in the terraform.tfvars

export SUBSCRIPTION_ID=
export RESOURCE_GROUP_NAME=
export STORAGE_ACCOUNT_NAME=

Set azure connection

az login
az account set --subscription $SUBSCRIPTION_ID
az configure --defaults group=$RESOURCE_GROUP_NAME

Bootstrap terraform

# move to bootstrap directory
cd bootstrap
# initialize terraform
terraform init
# plan the infrastructure change, ignore the undeclared variable errors
terraform plan -var-file=../terraform.tfvars -out plan
# apply the infrastructure change. Will create a resource group and storage account inside the resource group
terraform apply plan

• Get the access key for the storage account and store it in your environment as "ARM_ACCESS_KEY".

Setting the Azure RM Access Key

export ARM_ACCESS_KEY=`az storage account keys list --resource-group $RESOURCE_GROUP_NAME --account-name $STORAGE_ACCOUNT_NAME --query [0].value | tr -d '"'`

Creating a Cluster

As soon as you have a storage account, you can create the Kubernetes cluster. This will also create a MariaDB Azure server, in accordance with the configuration in the db.tf and db_config.tf files.

• To create the cluster and the container registry, run the commands in the next code block:

# return to main terraform directory
cd ..

# initialize terraform
terraform init -backend-config="access_key=$ARM_ACCESS_KEY" \
 -backend-config="storage_account_name=$STORAGE_ACCOUNT_NAME" \
 -backend-config="resource_group_name=$RESOURCE_GROUP_NAME"

# inspect changes
terraform plan -out clusterplan

# apply changes, this will output the randomly generated database password
# it can happen that the created service principal is not ready yet for the cluster, which can result in an error, in this case simply run the command again
terraform apply clusterplan

• The command terraform apply generates a random password for the database, outputted to the console.

Cluster creation on Azure might take up to a quarter of an hour. After this, the running costs of your subscription will increase, because of the creation of various infrastructure resources.

Some considerations:

• If you are setting up an environment that you plan to keep for a longer time, consider putting the various files and settings into a version control system such as Git.
• To better record your infrastructure as code, consider setting your choices as defaults in the Terraform files, so that the state of your infrastructure is fully defined there.

Connect to the Kubernetes cluster

az aks get-credentials --resource-group $RESOURCE_GROUP_NAME --name aks-cluster --overwrite-existing