Skip to main content
Version: 4.5.x LTS

Kubernetes-Based Installation

EXPERIMENTAL FEATURE

  • Experimental means that various things will change and that the Kubernetes integration is not yet supported for production use.
  • Production support is planned from the next rolling release onward (February 2020, expected date as of November 2019).
  • The experimental Kubernetes functionality will not be maintained in the November 2019 LTS release.
  • The goal is to get early feedback from you and give you early access to the upcoming Kubernetes solution.
  • We expect the following modifications and enhancements:
    • In this release, multiple replicas for nevisAuth and nevisIDM are not supported. This is because they require sticky sessions for most use cases.

Introduction

If you want to deploy Nevis onto a Cloud infrastructure, we recommend using the Kubernetes-based installation. This installation is based on the Docker containerization and Kubernetes orchestration technologies. For a general overview of the Nevis-on-Kubernetes deployment solution, see Main Concepts.

In this installation tutorial, you will set up Kubernetes from scratch. This is a multi-step process.

Prerequisites

This documentation uses Terraform to manage the infrastructure on the cloud service provider. Terraform allows managing the infrastructure as code, which provides additional benefits such as version control.

The use of Terraform is not required.

This tutorial provides a way to set up a Kubernetes cluster on Azure. Any other cloud providers have not been tested and are not officially supported by Nevis at this time. We tested the instructions on RHEL 7.

A couple of things are required to get started with Nevis on Kubernetes:

  • Have an Azure subscription and have enough permissions to create resource groups and resources.
  • A Linux environment with the following software pre-installed:

Expand the panels for more details about tested versions of the tools.

Click here to expand...Terraform v0.12.7

  • provider.azurerm v1.33.1

  • provider.kubernetes v1.9.0

  • provider.random v2.2.0

  • provider.tls v2.1.

    Click here to expand...Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.1", GitCommit:"4485c6f18cee9a5d3c3b4e523bd27972b1b53892", GitTreeState:"clean", BuildDate:"2019-07-18T09:18:22Z", GoVersion:"go1.12.5", Compiler:"gc",Platform:"linux/amd64"}

    Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.7", GitCommit:"6f482974b76db3f1e0f5d24605a9d1d38fad9a2b", GitTreeState:"clean", BuildDate:"2019-03-25T02:41:57Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

    Click here to expand...Client:

  • Version: 1.13.1

  • API version: 1.26

  • Package version: docker-1.13.1-96.gitb2f74b2.el7.x86_64

  • Go version: go1.10.8

  • Git commit: b2f74b2/1.13.1

  • Built: Tue Apr 2 21:01:07 2019

  • OS/Arch: linux/amd64Server:

  • Version: 1.13.1

  • API version: 1.26 (minimum version 1.12)

  • Package version: docker-1.13.1-96.gitb2f74b2.el7.x86_64

  • Go version: go1.10.8

  • Git commit: b2f74b2/1.13.1

  • Built: Tue Apr 2 21:01:07 2019

  • OS/Arch: linux/amd64

  • Experimental: false

    Click here to expand...

  • azure-cli 2.0.70

This guide requires basic knowledge of Linux, Terraform, Kubernetes and the Azure CLI. In case you are new to these topics, we recommend that you see tutorials or courses available online. If you are low on time, focus on the Kubernetes tutorials, for example: Viewing Pods and Nodes.

Steps to Perform

Perform the following steps for a Kubernetes-based installation of nevisAdmin 4:

  1. [Setting Environment Variables]
  2. [Downloading and Adapting Template Files]
  3. [Creating the Kubernetes Cluster with Terraform]
  4. [Uploading Nevis Docker Images]
  5. [Deploying the nevisOperator]
  6. [Preparing the Git Deployment Repository]
  7. [Deploying nevisAdmin 4]
  8. [Preparing the Test Namespace for Nevis Deployment]
  9. [Configuring an Example Project and Inventory in the GUI]

The following chapters describe each step in detail.

Setting Environment Variables

First, set the next environment variables:

Environment Preconfiguration​

SUBSCRIPTION_ID=123ab45c-d56-78e9-0fg1-h2ijk34lmn5o
STORAGE_ACCOUNT_NAME=terraform123456789                  # globally unique and only lowercase alphanumeric characters allowed
RESOURCE_GROUP_NAME=nevis-cluster                        # unique within Azure subscription
CLUSTER_NAME=nevis-cluster
VNET_NAME="${RESOURCE_GROUP_NAME}-vnet"
LOCATION="West Europe"
KUBERNETES_VERSION="1.13.11"                             # Kubernetes version to use on the cluster (supported: 1.13.11+)
                                                         # Command to list available versions:
                                                         # az aks get-versions --location 'West Europe' -o table |grep ^1.13 |cut -f 1 -d" "
DNS_PREFIX=nevis-cluster                                 # unique within Azure region
ADMIN_USER=myownadmin
CONTAINER_REGISTRY_NAME=mydockerregistry                 # globally unique DNS label, only lowercase alphanumeric characters allowed /  the registry will be accessible at <CONTAINER_REGISTRY_NAME>.azurecr.io

Downloading and Adapting Template Files

All files required to set up Terraform and Kubernetes are provided in the following Zip file: nevis-cluster.zip

  • First, download and unzip the files. After unzipping, you will have the following files in the nevis-cluster directory (= the project directory):
FileDescription
terraform/bootstrap/terraform-storage.tfDefines the storage account used to store the state of the Terraform managed infrastructure.
terraform/aks-cluster.tfDefines the actual Kubernetes cluster.
terraform/container-registry.tfDefines a Docker container registry accessible in the cluster.
terraform/azure.tfLinks the Azure cluster/registry to a storage account.
kubernetes-operator/Configures the Custom Resource Definitions for Nevis and deploys the nevisOperator.
kubernetes-nevisadmin4/Deploys an example MariaDB that holds nevisAdmin 4 data, deploys nevisAdmin 4 itself, and deploys nginx to access nevisAdmin 4 from the public internet.
kubernetes-github-account/Configures a GitHub account (used as an example nevisAdmin 4 Git deployment repository).
kubernetes-test-namespace/Configures the Kubernetes namespace where the Nevis components will run. Deploys example MariaDBs used by the components. Configures nginx to expose nevisProxy.

Files in the nevis-cluster directory- Next, adapt the files as follows:

  1. Replace the default nevis-cluster value in the following file with your $RESOURCE_GROUP_NAME:
kubernetes-nevisadmin4/nginx.yaml:216:                        service.beta.kubernetes.io/azure-load-balancer-resource-group: nevis-cluster

  1. Replace the default nevis-cluster value in the following files with your $DNS_PREFIX:
kubernetes-nevisadmin4/nevisadmin4.yaml:96:                   - host: nevis-cluster.westeurope.cloudapp.azure.com
kubernetes-nevisadmin4/nevisadmin4.yaml:105:                  - nevis-cluster.westeurope.cloudapp.azure.com
kubernetes-test-namespace/ingress-proxy.yaml:9:               - host: nevis-cluster.westeurope.cloudapp.azure.com
  1. Replace the default value nevisregistry with your $CONTAINER_REGISTRY_NAME:
kubernetes-nevisadmin4/nevisadmin4.yaml:22:                   value: nevisregistry.azurecr.io
kubernetes-nevisadmin4/nevisadmin4.yaml:23:                   image: nevisregistry.azurecr.io/nevis/nevisadmin4-alpine:latest
kubernetes-operator/operator.yaml:237:                        image: nevisregistry.azurecr.io/nevis/nevisoperator:latest
kubernetes-operator/operator.yaml:171:                        container-registry: nevisregistry.azurecr.io

Creating the Kubernetes Cluster with Terraform

The next step is to create a Kubernetes cluster with Terraform. First bootstrap Terraform before you can create the cluster.

Bootstrapping Terraform​

Terraform can be used to easily set up a Kubernetes cluster and a Docker registry on Azure.

  • First, connect to Azure by running the following commands:

Obtaining the Azure RM Access Key​

az login
az account set --subscription $SUBSCRIPTION_ID
az configure --defaults group=$RESOURCE_GROUP_NAME
  • Now you can initialize a storage account to store the state of Terraform. Use the next commands:

Bootstrap terraform​

cd terraform/bootstrap                                 # move to boostrapping directory
terraform init                                         # initialize terraform

 # plan the infrastructure change
terraform plan -var "subscription=$SUBSCRIPTION_ID" \
  -var "location=$LOCATION" \
  -var "resource_group_name=$RESOURCE_GROUP_NAME" \
  -var "storage_account_name=$STORAGE_ACCOUNT_NAME"

 # apply the infrastructure change. Will create a resource group and storage account inside the resource group
terraform apply -var "subscription=$SUBSCRIPTION_ID" \
  -var "location=$LOCATION" \
  -var "resource_group_name=$RESOURCE_GROUP_NAME" \
  -var "storage_account_name=$STORAGE_ACCOUNT_NAME"

cd ..                                                  # return to main terraform directory
az storage account keys list \
    --resource-group $RESOURCE_GROUP_NAME \
    --account-name $STORAGE_ACCOUNT_NAME
  • The last command should return a list of keys. Copy the first value of the keys and store it as the ARM_ACCESS_KEY environment variable:

Setting the Azure RM Access Key​

ARM_ACCESS_KEY=<the value of the first key>

Creating a Cluster​

Now that the storage account exists, you can create the Kubernetes cluster. Before running the commands below, you can configure the cluster to your specific needs. To do this, specify the type and count of the nodes on which the cluster will run in the aks-cluster.tf file.

  • To create the cluster and the container registry, run the commands in the next code block:

Creating the Cluster​

# initialize terraform
terraform init -backend-config="access_key=$ARM_ACCESS_KEY" \
  -backend-config="storage_account_name=$STORAGE_ACCOUNT_NAME" \
  -backend-config="resource_group_name=$RESOURCE_GROUP_NAME"

# inspect changes
terraform plan -var "location=$LOCATION" \
  -var "resource_group_name=$RESOURCE_GROUP_NAME" \
  -var "name=$CLUSTER_NAME" \
  -var "vnet_name=$VNET_NAME" \
  -var "dns_prefix=$DNS_PREFIX" \
  -var "admin_user=$ADMIN_USER" \
  -var "kubernetes_version=$KUBERNETES_VERSION" \
  -var "registry_name=$CONTAINER_REGISTRY_NAME"
     
# apply changes                 
terraform apply -var "location=$LOCATION" \
  -var "resource_group_name=$RESOURCE_GROUP_NAME" \
  -var "name=$CLUSTER_NAME" \
  -var "vnet_name=$VNET_NAME" \
  -var "dns_prefix=$DNS_PREFIX" \
  -var "admin_user=$ADMIN_USER" \
  -var "kubernetes_version=$KUBERNETES_VERSION" \
  -var "registry_name=$CONTAINER_REGISTRY_NAME"

cd ..                                                  # return to main nevis-cluster directory
  • If the command terraform apply fails at some point, just relaunch it.

Cluster creation on Azure might take up to quarter of an hour. After this, the running costs of your subscription will increase, because various infrastructure resources will be created.

After performing the previous steps, a fully functional cluster and a Docker container registry should be available. The next step is to deploy the operator to the cluster.

Some considerations:

  • If you are setting up an environment that you plan to keep for a longer time, consider putting the various files and settings into a version control system such as Git.
  • To better record your infrastructure as code, consider setting your choices as defaults in the Terraform files, so that the state of your infrastructure is fully defined there. Uploading Nevis Docker Images

All Nevis Docker images needed to run Nevis on Kubernetes are provided via AdNovum Client Connect (Binary Repository for Deliveries).

The example below shows the commands we used to copy the Docker images from the AdNovum Client Connect registry to the Azure registry. Adapt the target registry (MY_REGISTRY) to your environment.

Copy the Docker images​

ADNOVUM_REGISTRY=nevis-docker-cc-binrepo.clientconnect.adnovum.net
MY_REGISTRY=nevisregistry

az acr login --name $MY_REGISTRY

declare -a images=("nevis/nevisproxy:3.14"
 "nevis/nevisproxy-dbschema:0.1"
 "nevis/nevislogrend:1.7"
 "nevis/nevisauth:4.25"
 "nevis/nevisauth-dbschema:0.1"
 "nevis/nevisidm:2.75"
 "nevis/nevisidm-dbschema:2.75"
 "nevis/nevisadmin4:4.5"
 "nevis/nevisoperator:4.5"
 "nevis/nevis-git-init:0.4")
for i in "${images[@]}"; do
 docker pull $ADNOVUM_REGISTRY/$i
 NAME=$(echo "$i" | cut -d '/' -f 2 | cut -d ':' -f 1 )
 docker tag $ADNOVUM_REGISTRY/$i $MY_REGISTRY.azurecr.io/$i
 docker push $MY_REGISTRY.azurecr.io/$i
 docker tag $ADNOVUM_REGISTRY/$i $MY_REGISTRY.azurecr.io/nevis/$NAME:latest
 docker push $MY_REGISTRY.azurecr.io/nevis/$NAME:latest
done

The version numbers above correspond to the November 2019 cluster release. Adapt the version numbers in case you are installing a newer release.

Deploying the nevisOperator

  • Configure kubectl to point to the newly created cluster:

Configuring kubectl​

az aks get-credentials --resource-group $RESOURCE_GROUP_NAME --name $CLUSTER_NAME --overwrite-existing
  • Apply the commands below from your project directory:

Deploy CRDs and Operator to the cluster​

# Nevis Custom Resource Definitions
kubectl apply -f kubernetes-operator/crd

# Install accessory resources needed by the operator and fire up the nevisOperator
kubectl apply -f kubernetes-operator/operator.yaml

# find out which pod is used for nevisoperator
kubectl get pods --all-namespaces | grep nevisoperator-controller-manager

# Check logs as follows
kubectl logs -n nevisoperator-system -c manager nevisoperator-controller-manager-<random_id>

Preparing the Git Deployment Repository

In this tutorial, we use GitHub as the Git system. nevisAdmin 4 currently only supports SSH-based Git access. This means that follow the next instructions to configure it.

  1. Prepare GitHub:

  2. Create a GitHub account.

  3. Create an empty Git repository called "deploy".

  4. Set the GitHub repository to "private": `http://help.github.com/en/articles/setting-repository-visibility#making-a-repository-private

  5. Create SSH key material:

  6. Follow the instructions on this page (you may skip the SSH agent): `http://help.github.com/en/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent

  7. For this tutorial, do not enter a passphrase.

  8. Add the key to your GitHub account:

  9. See `http://help.github.com/en/articles/adding-a-new-ssh-key-to-your-github-account

  10. Copy the content of the new private and public key files (for example, id_rsa and id_rsa.pub) to the nevisAdmin configurations. Create a tar package from the configurations:

# Copy the keys
cat id_rsa >> kubernetes-nevisadmin4/configs/var/opt/nevisadmin4/conf/ssh/key
cat id_rsa.pub >> kubernetes-nevisadmin4/configs/var/opt/nevisadmin4/conf/ssh/key.pub

# Create the tar package
tar -czf nevisadmin4.tar.gz -C kubernetes-nevisadmin4/configs/var/opt/nevisadmin4/ .
  1. Copy/paste the private key and the URL of your GitHub repository into the following file:
vi kubernetes-test-namespace/test-resources.yaml
# Notes:
# - replace the example url with your repository url
# - replace the existing example key with your own key
# - ssh_known_hosts is already set up for github.com

  1. Step Preparing the test namespace for Nevis deployment describes how to deploy the secrets. They will be used by the git-init container.
  2. Step Deploying nevisAdmin 4 explains how to register another key to GitHub. This key is meant for the nevisAdmin 4 connection.

Deploying nevisAdmin 4

The next steps will guide you through the deployment of nevisAdmin 4 and the database instance, along with their required configuration and service objects. You deploy the following Kubernetes objects:

  • nevisAdmin 4 DB
  • nevisAdmin 4
  • NGINX

You can verify if you are connected to the correct cluster with the kubectl tool:

kubectl config current-context

Deploying nevisAdmin 4 Component and Its DB​

We run nevisAdmin 4 in a separate namespace. The DB is required by nevisAdmin 4 to store application data, such as your configuration projects, inventories, and more.

  • First, run the following commands:

Apply the changes to the cluster​

# Create the namespace
kubectl apply -f kubernetes-nevisadmin4/nevisadmin4-namespace.yaml

# Configure and start the DB as follows:
kubectl apply -f kubernetes-nevisadmin4/nevisadmin4db.yaml

# Create a kubernetes secret from the previously created tar package containing the nevisAdmin4 configurations
kubectl create secret generic nevisadmin4-config-th74bkmgm7 --from-file=./nevisadmin4.tar.gz -n nevisadmin4

# Wait for about a minute for the DB to initialize.

# Now start nevisAdmin 4.
# The configuration files contained in the created secret above will be mounted inside the nevisadmin4 container.
kubectl apply -f kubernetes-nevisadmin4/nevisadmin4.yaml
  • Next, follow the steps mentioned in the code block below to validate your nevisAdmin4 deployment:

Validate the nevisAdmin4 deployment​

# Find out which pod is used for nevisadmin
kubectl get pods --all-namespaces | grep nevisadmin4-

# Update the pod name below and check the nevisAdmin4 logs
kubectl logs -n nevisadmin4 nevisadmin4-<random_id>

# Update the pod name in all of the commands below
# Check that the configurations and keymaterial have been mounted correctly
kubectl exec -it nevisadmin4 nevisadmin4-<random_id> ls -l /var/opt/nevisadmin4/conf/
kubectl exec -it nevisadmin4 nevisadmin4-<random_id> cat /var/opt/nevisadmin4/conf/nevisadmin4.yaml
kubectl exec -it nevisadmin4 nevisadmin4-<random_id> ls -l /var/opt/nevisadmin4/conf/ssh/
kubectl exec -it nevisadmin4 nevisadmin4-<random_id> cat /var/opt/nevisadmin4/conf/ssh/key
kubectl exec -it nevisadmin4 nevisadmin4-<random_id> cat /var/opt/nevisadmin4/conf/ssh/key.pub

Deploying NGINX Ingress​

To expose the Nevis setup outside the cluster, two options are possible:

  • Use an Ingress. This option is the right choice if you want to use Ingress functionality, such as routing traffic to services other than nevisProxy, e.g., nevisAdmin 4. It is also suitable if you want to use let's encrypt.
  • Use a simple load balancer. The load balancer setup is currently not explained in this tutorial. Use this if:

For the Ingress controller (and actually also for the simple load balancer), a static IP needs to be configured in Azure. This was already done via Terraform on the cluster setup. This static IP needs to be configured on line 223 of the following file. The IP address can be obtained via the following ways:

  • az network public-ip list
  • terraform show and use the value of aks_cluster_ingress_ip.
# Update this file to the IP address that Azure allocated.
kubernetes-nevisadmin4/nginx.yaml:223: loadBalancerIP: 192.0.2.1

Now, start the NGINX Ingress:

Deploy the NGINX Ingress-controller and the necessary roles and role bindings​

kubectl apply -f kubernetes-nevisadmin4/nginx.yaml

After some time, you will be able to access the GUI at the following URL:

google-chrome https://$CLUSTER_NAME.westeurope.cloudapp.azure.com/nevisadmin/
# login as admin/welcome2nevis and change your password immediately

Change your password immediately after logging in. This prevents unauthorized access.

Preparing the Test Namespace for Nevis Deployment

The next step is to configure the Kubernetes namespace where the Nevis components will run. It contains several example MariaDBs that will be used by the components. Also, NGINX is configured to expose nevisProxy.

kubectl apply -f kubernetes-test-namespace/test-namespace.yaml
kubectl apply -f kubernetes-test-namespace/test-resources.yaml -n test
kubectl apply -f kubernetes-test-namespace/ingress-proxy.yaml -n test

Configuring an Example Project and Inventory in the GUI

The final step is to import and deploy an example project using nevisAdmin 4:

  1. From the nevisAdmin 4 welcome screen, import the following project: project_CLOUD-PROJECT_20191117T113222Z.zip
  2. In the Administration>Inventory Settings screen, import the following inventory: inventory_CLOUD-INVENTORY_20191117T122521Z.zip Adapt the inventory:
  • Enter the URL of your GitHub repository.
  • Enter the URL and token of your Kubernetes cluster:

Get the token of your Kubernetes cluster​

# Find the name of the secret containing the token
kubectl get secret -n default | grep default

# Use the secret name from the output from the command above
kubectl describe secret -n default default-token-<random_id> | grep token
  1. Click the Deploy button and go through the steps of the Deployment Wizard. If there is a warning, you can just accept it and continue.
  2. Go to the following URLs to test:
# Example protected application
google-chrome https://$CLUSTER_NAME.westeurope.cloudapp.azure.com/teletext/

# Example standalone nevisAuth flow (not actually logging in because nevisIDM is not available, yet)
google-chrome https://$CLUSTER_NAME.westeurope.cloudapp.azure.com/hello-auth/