Creating Kubernetes Cluster on Azure

In this tutorial, you will set up a Kubernetes cluster from scratch on Azure with the dependencies Nevis needs. For a general overview of the Nevis-on-Kubernetes deployment solution, see Kubernetes Deployment (Cloud).

For an easier installation process, check Azure deployment automation, where the cluster is set up automatically with nevisAdmin4 already running in a few minutes.

The tutorial creates the following resources in Azure:

• A Terraform storage account, which will hold our cluster configuration.
• A Kubernetes cluster.
• An Azure Database for PostgreSQL flexible server, on which we will create our databases containing the Nevis-related data.
• A container registry, which will hold our docker images.
• A virtual network, which the cluster will use.
• Two IP addresses, one to access the cluster, and one to the cluster itself.

See Restrictions in Kubernetes Setups, which includes limitations specific to Kubernetes.

Prerequisites

This documentation uses Terraform to manage the infrastructure on the cloud service provider. Terraform allows managing the infrastructure as code, which provides additional benefits such as version control.

A couple of things are required to get started with Nevis on Kubernetes:

• Have an Azure subscription and have enough permissions to create resource groups and resources. This includes Application Administrator for creating the service principal, and Owner to assign the required Role to it.
• The supported Kubernetes versions for this guide are listed in the Kubernetes versions support policy.
• A Linux environment with the following software pre-installed:
  • terraform: Terraform command line tool.
  • az: Azure command line interface.

This guide requires basic knowledge of Linux, Terraform and the Azure CLI. In case you are new to these topics, we recommend to use the Azure deployment automation instead.

Downloading Template Files

All files required to set up the Kubernetes cluster are provided in the terraform/aks-cluster-setup directory of the nevis-kubernetes-support repository.

First, clone the repository, or download the terraform/aks-cluster-setup directory.

File	Description
bootstrap/terraform-storage.tf	Defines the storage account used to store the state of the Terraform managed infrastructure.
aks-cluster.tf	Defines the actual Kubernetes cluster.
container-registry.tf	Defines a Docker container registry accessible in the cluster.
azure.tf	Points the Terraform state at the storage account.
providers.tf	Declares the required Terraform providers.
db.tf	Defines the Azure Database for PostgreSQL flexible server.
db_config.tf	Configures the Azure PostgreSQL server.
variables.tf	Contains the variables used in the template files.
terraform.tfvars	Contains the variable values.
validate-tfvars.sh	Validates terraform.tfvars against the naming rules Azure enforces, without requiring terraform init.

Setting Terraform Variables

Most values in the provided terraform.tfvars file already have working defaults. The following ones are left empty and must be filled in, since Terraform cannot pick a sensible default for them — either they need to be globally unique across Azure, or they identify your specific subscription:

Variable	Why it's required
subscription	Your Azure subscription ID. Determines which subscription every resource is created in. Find it with az account show --query id.
resource_group_name	Name of the main resource group Terraform creates, holding almost every resource (storage account, AKS cluster, container registry, PostgreSQL server, virtual network, public IP). Must be unique within your subscription.
node_resource_group_name	AKS always places the node pool's VMs, disks, and networking in a second, separate resource group. This names it explicitly; otherwise Azure assigns an auto-generated name such as MC_<resource_group_name>_<cluster_name>_<location>. Must be unique within your subscription, and different from resource_group_name.
storage_account_name	Name of the storage account created to hold the Terraform remote state (not a Nevis resource). Azure Storage Account names must be globally unique across all of Azure, 3-24 characters, lowercase letters and numbers only.
cluster_name	Name of the AKS cluster resource, also used to derive the virtual network and subnet names. Must be unique within your subscription. Recommended to match the resource group name.
registry_name	Name of the Azure Container Registry Terraform creates for you (not a reference to an existing one). The Nevis Docker images are pushed to and pulled from this registry later, during the nevisAdmin4 installation. Must be globally unique across Azure, alphanumeric characters only.
dns_prefix	Prefix for the AKS API server's public hostname (<dns_prefix>-<random-id>.hcp.<location>.azmk8s.io). This is only the Kubernetes API server's DNS name, not the URL where nevisAdmin4 or the Nevis applications will be reachable — that domain is configured separately, later, during the nevisAdmin4 installation. Must be unique within the Azure region.
db_server	Name of the Azure Database for PostgreSQL flexible server created to host the Nevis databases (nevisIDM, nevisAdmin4, etc.). Must be globally unique across Azure, since it becomes part of the server's public DNS name (<db_server>.postgres.database.azure.com).
db_root_user	Administrator login name for the PostgreSQL server above — the DB_ROOT_USER used later when preparing the database credential secret during the nevisAdmin4 installation. Cannot be a reserved name such as "root" or "azure_superuser".

The template validates the format of these values (length, allowed characters, reserved names) when you run terraform plan. To check this upfront without running terraform init first, run the provided script instead:

./validate-tfvars.sh terraform.tfvars

Neither the script nor terraform plan can check whether a name that has to be globally unique across Azure is actually still available — that only happens once Terraform tries to create the resource. To catch a naming conflict before running terraform apply, check availability upfront:

az storage account check-name-availability --name <storage_account_name>
az acr check-name --name <registry_name>

For db_server, cluster_name, and dns_prefix, there is no dedicated availability-check command; if terraform apply fails because one of these is already taken, pick a different value and re-run it.

Creating the Kubernetes Cluster with Terraform

The next step is to create a Kubernetes cluster with Terraform. Bootstrap Terraform before you can create the cluster.

Bootstrapping Terraform

Terraform can be used to easily set up a Kubernetes cluster and a Docker registry on Azure. Perform the next steps/execute the following commands:

Set the following environment variables, use the same values that were used in the terraform.tfvars

export SUBSCRIPTION_ID=
export RESOURCE_GROUP_NAME=
export STORAGE_ACCOUNT_NAME=
export CLUSTER_NAME=
export DB_SERVER=

Set azure connection

az login
az account set --subscription $SUBSCRIPTION_ID
az configure --defaults group=$RESOURCE_GROUP_NAME

Bootstrap terraform

# move to bootstrap directory
cd bootstrap
# initialize terraform
terraform init
# plan the infrastructure change, ignore the undeclared variable errors
terraform plan -var-file=../terraform.tfvars -out plan
# apply the infrastructure change. Will create a resource group and storage account inside the resource group
terraform apply plan

Setting the Azure RM Access Key

Get the access key for the storage account and store it in your environment as ARM_ACCESS_KEY:

export ARM_ACCESS_KEY=`az storage account keys list --resource-group $RESOURCE_GROUP_NAME --account-name $STORAGE_ACCOUNT_NAME --query [0].value | tr -d '"'`

Creating a Cluster

As soon as you have a storage account, you can create the Kubernetes cluster. This will also create an Azure Database for PostgreSQL flexible server, in accordance with the configuration in the db.tf and db_config.tf files.

PostgreSQL version

db.tf pins a specific PostgreSQL major version (currently 17). Azure automatically applies minor/patch updates within that major version, but never upgrades the major version itself — that always requires an explicit, separate migration step. Check the Nevis Product Lifetime and Platform Support Matrix for the PostgreSQL versions supported by your Nevis version, and keep db.tf in sync if that changes.

The pinned azurerm provider version in providers.tf also has to support the target PostgreSQL version — the provider rejects unrecognized version strings at plan time. When bumping the PostgreSQL version, stay within the same azurerm major version if possible (a major provider version bump, e.g. 4.x to 5.x, can include breaking changes elsewhere in this template).

Database network access

This template does not set up VNet integration for the database, so db.tf enables public network access on the PostgreSQL server, restricted by a firewall rule to only the AKS cluster's own egress IP (the aks_egress_ip output from aks-cluster.tf). This is not open to the internet, but it is still public access, not a private network path. For a production setup, replace this with VNet integration (a delegated subnet and private DNS zone) instead.

Allow-list the citext extension (only if not pre-provisioning it yourself)

Azure Database for PostgreSQL flexible server blocks CREATE EXTENSION for extensions that aren't explicitly allow-listed, regardless of the role's privileges. The nevisAdmin4 dbschema migrations need the citext extension, which isn't allow-listed by default. If you'd rather have the dbschema job create it (using azure_pg_admin membership), allow-list it once, after the database is created:

az postgres flexible-server parameter set --resource-group $RESOURCE_GROUP_NAME \
  --server-name $DB_SERVER --name azure.extensions --value citext

If you're using a minimal-permission database.root role instead, you can skip this and create citext yourself as the real admin — see the "Minimal permissions for database.root" note in Installation.

• To create the cluster and the container registry, run the commands in the next code block:

# return to main terraform directory
cd ..

# initialize terraform
terraform init -backend-config="access_key=$ARM_ACCESS_KEY" \
 -backend-config="storage_account_name=$STORAGE_ACCOUNT_NAME" \
 -backend-config="resource_group_name=$RESOURCE_GROUP_NAME"

# inspect changes
terraform plan -out clusterplan

# apply changes, this will output the randomly generated database password
# it can happen that the created service principal is not ready yet for the cluster, which can result in an error, in this case simply run the command again
terraform apply clusterplan

• The command terraform apply generates a random password for the database, outputted to the console.

Cluster creation on Azure might take up to a quarter of an hour. After this, the running costs of your subscription will increase, because of the creation of various infrastructure resources.

Some considerations:

• If you are setting up an environment that you plan to keep for a longer time, consider putting the various files and settings into a version control system such as Git.
• To better record your infrastructure as code, consider setting your choices as defaults in the Terraform files, so that the state of your infrastructure is fully defined there.

Stopping and Starting the Cluster

The cluster runs 24/7 by default: the node pool is a fixed size, with no auto-scaling or scheduled shutdown configured. If you don't need the cluster continuously, you can pause it without destroying it:

az aks stop --resource-group $RESOURCE_GROUP_NAME --name $CLUSTER_NAME
az postgres flexible-server stop --resource-group $RESOURCE_GROUP_NAME --name $DB_SERVER

This stops billing for the AKS control plane, node VMs, and the PostgreSQL server. The container registry and storage account are not affected, as they are not compute resources and have no running state to stop.

To resume, start the database first, so it's ready before the Nevis components in the cluster try to connect to it:

az postgres flexible-server start --resource-group $RESOURCE_GROUP_NAME --name $DB_SERVER
az aks start --resource-group $RESOURCE_GROUP_NAME --name $CLUSTER_NAME

info

The state of a stopped AKS cluster is only preserved for up to 12 months. If it stays stopped longer than that, the state can no longer be recovered.

If you are done with the cluster for good, delete all created resources instead by running terraform destroy from the main aks-cluster-setup directory.

Connect to the Kubernetes cluster

az aks get-credentials --resource-group $RESOURCE_GROUP_NAME --name $CLUSTER_NAME --overwrite-existing

This adds a context for the cluster to your kubeconfig and switches to it. Confirm you're pointed at the right cluster and that it's healthy before continuing:

# Confirm the current context matches your cluster name
kubectl config current-context

# The API server should respond, and all nodes should show status "Ready"
kubectl cluster-info
kubectl get nodes

# Confirm the node count matches num_agents in terraform.tfvars
kubectl get nodes --no-headers | wc -l

# Only system pods should exist at this point (kube-system, etc.); this cluster
# doesn't have nevisAdmin4 or any Nevis components deployed yet
kubectl get pods -A

If kubectl get nodes doesn't return the expected number of nodes as Ready, wait a minute and retry — nodes can take a short while to join after the cluster finishes provisioning.