Installation on supported platforms

See the [Recommendations and Limitations] chapter for limitations specific to Kubernetes.

Introduction

If you want to deploy Nevis onto a Cloud infrastructure, we recommend using a Kubernetes-based installation. This installation is based on the Docker containerization and Kubernetes orchestration technologies. For a general overview of the Nevis-on-Kubernetes deployment solution, see Main Concepts.

• In the chapter Kubernetes Deployment Troubleshooting you can find information about how to debug the configuration.
• In the chapter Kubernetes Upgrade you can find information about how to upgrade your existing installation.

For limitations specific to Kubernetes, see Recommendations and Limitations.

In this installation tutorial, you will set up nevisAdmin4 on an existing cluster.

Installing on supported Kubernetes platforms

Pay attention to the following points:

• For general guidance, refer to System Requirements.

• For the database, you should use a MariaDB server.

• In some Kubernetes installations, special permission settings or entitlements are required, for example, for the nevisOperator component.

• Any Kubernetes-compatible product or cloud provider should work.

  • For more details about the supported versions of the above providers, see the "Nevis Product Lifetime and Platform Support Matrix". Prerequisites

• Have an existing Kubernetes cluster and have enough permissions to create resource groups and resources, which includes RBAC permissions.

• Have a MariaDB database with the correct configuration, see component specific documentation such as: [Database Preparing#DatabaseConfiguration.1], generally the following is needed:

autocommit=0
transaction-isolation = READ-COMMITTED
log_bin_trust_function_creators = 1
lower_case_table_names = 1
character-set-server = utf8mb4

• The supported Kubernetes versions for this guide are 1.24, 1.25, 1.26, 1.27 and 1.28.
• A Linux environment with the following software pre-installed:

Expand the panels for more details about tested versions of the tools.

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4", GitCommit:"e6c093d87ea4cbb530a7b2ae91e54c0842d8308a", GitTreeState:"clean", BuildDate:"2022-02-16T12:38:05Z", GoVersion:"go1.17.7", Compiler:"gc", Platform:"linux/amd64"}

Client: Docker Engine - Community Version: 20.10.3

This guide requires basic knowledge of Linux and Kubernetes. In case you are new to these topics, we recommend that you see tutorials or courses available online. If you are low on time, focus on the Kubernetes tutorials, for example: Viewing Pods and Nodes.

Steps to Perform

Perform the following steps for a Kubernetes-based installation of nevisAdmin 4:

1. [Downloading Template Files]
2. [Preparing the Git Deployment Repository]
3. [Setting Environment Variables]
4. [Creating the Kubernetes Cluster with Terraform]
5. [Uploading Nevis Docker Images]
6. [Deploying the nevisOperator]
7. [Deploying nevisAdmin 4]
8. [Preparing the Test Namespace for Nevis Deployment]
9. [Configuring an Example Project and Inventory in the GUI]

The following chapters describe each step in detail.

Download Template Files

All files required to set up Terraform and Kubernetes are provided in the following Zip file: kubernetes-guide.zip

• First, download and unzip the files. The setup uses Gradle to easily customize the files based on environmental variables. The configuration files used by the setup can be found in kubernetes-guide/src/nevis-cluster.

File	Description
kubernetes-operator	Configures the Custom Resource Definitions for Nevis and deploys the nevisOperator.
kubernetes-nevisadmin4	Deploys nevisAdmin 4 itself, and deploys nginx to access nevisAdmin 4 from the public internet.
kubernetes-test-namespace/	Configures the Kubernetes namespace where the Nevis components will run. Configures nginx to expose nevisProxy.
kubernetes-cert-manager	Deploys a certificate manager that uses Let's Encrypt to get valid certificate for the running components.
kubernetes-nginx	Deploys NGINX.

Files in the nevis-cluster directory

Prepare the Git Deployment Repository

In this tutorial, we use GitHub as the Git system. However, it is possible to use Bitbucket, Gitea, GitLab, and so on, instead, the same options should be available in every case.

The nevisOperator currently only supports SSH-based Git access. Follow the next instructions to configure the Git connection for nevisOperator.

1. Prepare GitHub:
2. Create a GitHub account.
3. Create an empty Git repository called deploy.
4. Set the GitHub repository to private, refer to `http://help.github.com/en/articles/setting-repository-visibility#making-a-repository-private.
5. Create SSH key material:

Generating key material

# move into the main folder
cd kubernetes-guide
                                                   
ssh-keygen -t ecdsa -C "kubernetes" -m PEM -P "" -f id_rsa

# make sure ssh keys were generated
cat id_rsa
cat id_rsa.pub

This key pair will be read by Gradle and used during the setup process. Both the nevisOperator and nevisAdmin 4 will use this to connect go GitHub. 3. Add the key (id_rsa.pub) to your GitHub account, or to the repository itself as a deploy key:

1. Refer to `http://help.github.com/en/articles/adding-a-new-ssh-key-to-your-github-account.
2. Refer to`http://developer.github.com/v3/guides/managing-deploy-keys/ (the "Deploy keys" section).
3. The entry for GitHub is added by default. If you use a Git system other than GitHub, add the corresponding known_hosts entry to the following files:

kubernetes-guide/src/nevis-cluster/kubernetes-nevisadmin4/configs/var/opt/nevisadmin4/conf/ssh/known_hosts
kubernetes-guide/src/nevis-cluster/kubernetes-component-namespace/component-resources.yaml

The fingerprint can be found by using ssh-keyscan:

ssh-keyscan <host>

Set Environment Variables

First, fill out the following variables in the provided kubernetes-guide/environment file.

Environment variables

#!/bin/bash
# the domain used for the installation
# example nevis.test.com
export DOMAIN

# Information used for the Certificate Signing Request in automatic key management, by the nevisOperator
# example CH
export CSR_COUNTRY
# example Zurich
export CSR_PROVINCE
# example Zurich
export CSR_LOCALITY
# example Nevis Security AG
export CSR_ORGANIZATION
# example Nevis
export CRS_ORGANIZATIONAL_UNIT
# example [email protected]
export CSR_EMAIL_ADDRESS

# database server url, without port
# example: test.mariadb.database.azure.com
export DB_SERVER_URL

# database port
# example: 3306
export DB_SERVER_PORT

# user name of the root user for the database server
# example: dbroot
export DB_ROOT_USER

# DB root password
export DB_ROOT_PASSWORD

# the database user that will be created for admin4, to do the schema updates in the database
# example: schemauser
export ADMIN4DB_SCHEMA_USER

# the password for the schema user
export ADMIN4DB_SCHEMA_PASSWORD

# the database user that will be created for admin4, to connect to the database during operation
# example: applicationuser
export ADMIN4DB_APP_USER

# the password for the application user
export ADMIN4DB_APP_PASSWORD= 

# container registry the cluster uses, this is where the images will be pushed, and pulled from
# example: testnevisclusterr.azurecr.io, ghcr.io etc.
export CONTAINER_REGISTRY

# GIT deployment repository
# example: [email protected]:nevis/deploy.git
export GIT_URL

Generate configuration

Produce a customized configuration based on the environment variables:

Bootstrap

source environment
./gradlew bootstrap

Upload Nevis Docker Images

Use the provided script kubernetes-guide/publish_images.sh to copy the Docker images from the Nevis Portal registry to the Azure registry. Select one of the docker images in the portal under `http://portal.nevis.net/portal/secure/releases/rolling to acquire a temporary username/password for the registry.

This will transfer the following images:

Docker images

Run the script to transfer the images:

./publish_images.sh

Deploy NGINX Ingress

To expose the Nevis setup outside the cluster, an Ingress Controller is used. This also allows the use of Let's Encrypt to get valid certificates.

An existing NGINX ingress controller can also be used. If you do not have one available, deploy the NGINX Ingress with the provided manifest:

Deploy the NGINX Ingress controller and the necessary roles and role bindings

cd build/nevis-cluster # go back to the configuration folder
kubectl apply -f kubernetes-nginx/nginx.yaml

Or follow the official guide: `http://kubernetes.github.io/ingress-nginx/deploy/

Make sure the specified DOMAIN is pointing to the External IP of the NGINX Loadbalancer. To get the External IP check the created service:

kubectl get service -n ingress-nginx ingress-nginx

On some installations the snippet feature is disabled by default, as this is required for the side-by-side deployment to function, make sure you have the following in the ConfigMap used by ingress-nginx:

Config

allow-snippet-annotations: "true"
annotation-value-word-blocklist: load_module,lua_package,_by_lua,location,root,proxy_pass,serviceaccount,',\

It's recommended to use a blocklist to prevent possible misuse.

(info) Additional steps required for OpenShift

As NGINX must run with user 101 and need capabilities such as NET_BIND_SERVICE, on OpenShift changing the used SecurityContext might be required.

# edit the scc
kubectl edit scc -n ingress-nginx privileged

And extend the user list as follows:

users:
- system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount

For more information see: `http://cloud.redhat.com/blog/managing-sccs-in-openshift

Deploy the Certificate Manager

The setup uses Let's Encrypt to set a valid certificate. To make this work, you need to deploy a certificate manager on the cluster. Use the provided manifest:

Deploy the Certificate Manager

# in kubernetes-guide/build/nevis-cluster, use the following:
kubectl apply -f kubernetes-cert-manager/manager.yaml

Or follow the official guide: `http://cert-manager.io/docs/installation/

Deploy the issuers used by nevisAdmin4:

Deploy the issuers

# in kubernetes-guide/build/nevis-cluster, use the following:
# might have to wait a minute for the cert-manager to start before the issuers are successfully created
kubectl apply -f kubernetes-cert-manager/issuers.yaml

Deploy the nevisOperator

1. (Optional) If used on a cluster with multiple Ingress controllers, or with a different ClusterIssuer than the one provided in this guide, configure the following additional properties in the kubernetes-operator/operator.yml file as part of the nevisoperator-operator-config-6fc9hfhdhk ConfigMap, which will be used by the deployed nevisOperator.

Property	Default	Usage
ingress-class	-	Ingress class of the controller, see: `http://kubernetes.github.io/ingress-nginx/user-guide/multiple-ingress/
cluster-issuer	letsencrypt-prod	ClusterIssuer or Issuer that handles the certificate creation for the Ingress.
cluster-issuer-annotation	http://certmanager.k8s.io/cluster-issuer cert-manager.io/cluster-issuer	Annotation to specify the ClusterIssuer on the Ingress. By default both ClusterIssuer annotation is used for backwards compatibility. Specify it, if an Issuer is used instead.

1. (Optional) By default the operator is configured to watch all namespaces, this can be changed by by using the restrict-namespaces in kubernetes-operator/operator.yml, this also makes it possible to restrict the needed RBAC permissions:

restrict-namespaces:
  - namespace1
  - namespace2

3. Apply the commands below from the main directory:

Deploy CRDs and Operator to the cluster

# move in the nevis-cluster folder
# Create the operator namespace
kubectl apply -f kubernetes-operator/operator-namespace.yaml

# Nevis Custom Resource Definitions
kubectl apply -f kubernetes-operator/crd

# Create the ConfigMap containing the nevisOperator configuration properties
kubectl apply -f kubernetes-operator/operator-config.yaml

# Create the RBAC permissions needed by the operator
kubectl apply -f kubernetes-operator/operator-rbac.yaml

# Deploy by the operator
kubectl apply -f kubernetes-operator/operator.yaml

4. Check if everything went well:

Check status

# find out which pod is used for nevisoperator
kubectl get pods --all-namespaces | grep nevisoperator-controller-manager

# Check logs as follows
kubectl logs -n nevisoperator-system -c manager nevisoperator-controller-manager-<random_id>

Deploy nevisAdmin 4

The next steps guide you through the deployment of nevisAdmin 4. This includes the generation of the required configuration and service objects. You will deploy the following Kubernetes objects:

• nevisAdmin 4

To deploy the nevisAdmin 4 component, perform the steps below.

Note that nevisAdmin 4 runs in a separate namespace. The database is required by nevisAdmin 4 to store application data, such as your configuration projects, inventories, and more. ### OpenShift related adjustment

The nevisAdmin4 deployment specifies an fsGroup to have proper access for the attached volumes. Depending on the used Security Context, this might be restricted on OpenShift to certain ID-s, for more information see: `http://docs.openshift.com/container-platform/4.9/authentication/managing-security-context-constraints.html#authorization-SCC-strategies_configuring-internal-oauth

If this is the case, adjust the used fsGroup, this should only be necessary if a custom Security Context is used as the default "restricted" allows any ID:

• Adjust the used fsGroup in kubernetes-guide/src/nevis-cluster/kubernetes-nevisadmin4/nevisadmin4.yaml:

spec:
  securityContext:
    runAsNonRoot: true
    fsGroup: 2000

• Generate the build folder again:

# in the main folder
./gradlew bootstrap

Step 1: (Optional) Use one-way TLS for the database connection

To enable server authentication via a server certificate, the following steps are required:

1. Create a Java KeyStore (JKS) file containing the server certificate.

2. Store the JKS file under kubernetes-guide/src/nevis-cluster/kubernetes-nevisadmin4/configs/sshA keytool example to create the keystore:

keytool -importcert -file "BaltimoreCyberTrustRoot.crt.pem" -keystore mariadb.jks -alias "mariadb"

3. Adapt the database URL in the file kubernetes-guide/src/nevis-cluster/kubernetes-nevisadmin4/configs/config/nevisadmin4.yml, to make it use the created keystore file:

url: "jdbc:mysql://@DB_SERVER_URL@:@DB_SERVER_PORT@/nevisadmin4?useSSL=true&trustStore=/var/opt/nevisadmin4/conf/ssh/mariadb.jks&trustStorePassword=<password>"

4. For the database migration to also use SSL, extend the job definition nevisadmin4-dbschema and the nevisadmin4-dbschema-wait init container of the nevisadmin4 StatefulSet in the file kubernetes-guide/src/nevis-cluster/kubernetes-nevisadmin4/nevisadmin4.yaml with the following env variable:

env:
  - name: NEVIS_DBSCHEMA_USE_SSL
    value: "true"

5. Generate the build folder again:

# in the main folder
./gradlew bootstrap

Step 2: (Optional) Establish a secure connection between NGINX and nevisAdmin 4

The internal communication between NGINX and nevisAdmin 4 uses HTTP by default. To use HTTPS, perform the next steps:

1. Store the keystore containing the certificate under kubernetes-guide/src/nevis-cluster/kubernetes-nevisadmin4/configs/ssh. As the certificate is only used for securing internal traffic, it will not be visible outside the cluster.

The next code snippet shows how to create a self-signed certificate:

openssl req -x509 -newkey rsa:4096 -keyout myKey.pem -out cert.pem -days 3650
openssl pkcs12 -export -out keystore.p12 -inkey myKey.pem -in cert.pem -name nevisadmin

2. Extend the file kubernetes-guide/src/nevis-cluster/kubernetes-nevisadmin4/configs/config/nevisadmin4.yml to use the provided keystore as well as the right port:

server:
 port: 8443
 tls:
 keystore: /var/opt/nevisadmin4/conf/ssh/keystore.p12
 keystore-passphrase: ${env:NEVIS_KEYSTORE_PASSPHRASE}
 keystore-type: pkcs12
 key-alias: nevisadmin

3. Create the secret that contains the passphrase

kubectl create namespace nevisadmin4
kubectl create secret generic -n nevisadmin4 nevisadmin4-tls --from-literal=passphrase=<your_passphrase>

4.Generate the build folder again:

# in the main folder
./gradlew bootstrap

Step 3: Apply the changes to the cluster

Apply the changes to the cluster

# in kubernetes-guide/build/nevis-cluster, use the following:
# Create the namespace
kubectl apply -f kubernetes-nevisadmin4/nevisadmin4-namespace.yaml

# Create the configmap with the config files
kubectl create -n nevisadmin4 configmap nevisadmin4-config --from-file=kubernetes-nevisadmin4/configs/config/

# Create a kubernetes secret from the key materials used by admin4
kubectl create secret generic -n nevisadmin4 nevisadmin4-ssh --from-file=kubernetes-nevisadmin4/configs/ssh/

# Create a kubernetes secret with the environment variables used by admin4
kubectl create secret generic -n nevisadmin4 nevisadmin4-env --from-literal=rootpw=$DB_ROOT_PASSWORD --from-literal=schemauserpw=$ADMIN4DB_SCHEMA_PASSWORD --from-literal=applicationuserpw=$ADMIN4DB_APP_PASSWORD

# Now start nevisAdmin 4.
kubectl apply -f kubernetes-nevisadmin4/nevisadmin4.yaml

If HTTPS was enabled in Step 2, apply the following changes:

Apply HTTPS patch

# Apply only, if https is configured for nevisAdmin 4
kubectl apply -f kubernetes-nevisadmin4/nevisadmin4-https-patch.yaml

Step 4: Validate your nevisAdmin 4 deployment

Validate the nevisAdmin4 deployment

# Find out which pod is used for nevisadmin
kubectl get pods --all-namespaces | grep nevisadmin4-

# Update the pod name below and check the nevisAdmin4 logs
kubectl logs -n nevisadmin4 nevisadmin4-<random_id>

# Update the pod name in all of the commands below
# Check that the configurations and keymaterial have been mounted correctly
kubectl exec -it -n nevisadmin4 nevisadmin4-<random_id> -- ls -l /var/opt/nevisadmin4/conf/
kubectl exec -it -n nevisadmin4 nevisadmin4-<random_id> -- cat /var/opt/nevisadmin4/conf/nevisadmin4.yml
kubectl exec -it -n nevisadmin4 nevisadmin4-<random_id> -- ls -l /var/opt/nevisadmin4/conf/ssh/
kubectl exec -it -n nevisadmin4 nevisadmin4-<random_id> -- cat /var/opt/nevisadmin4/conf/ssh/key
kubectl exec -it -n nevisadmin4 nevisadmin4-<random_id> -- cat /var/opt/nevisadmin4/conf/ssh/key.pub

After some time, you will be able to access the GUI at the following URL:

google-chrome https://$DOMAIN/nevisadmin/
# login as admin/welcome2nevis and change your password immediately

Change your password immediately after logging in. This prevents unauthorized access.

Prepare the Component Namespace for Nevis Deployment

The next step is to configure the Kubernetes namespace where the Nevis components will run. NGINX is configured to expose nevisProxy.

# in kubernetes-guide/build/nevis-cluster, use the following:
kubectl apply -f kubernetes-component-namespace/component-namespace.yaml
kubectl apply -f kubernetes-component-namespace/component-resources.yaml -n component

Configure an Example Project and Inventory in the GUI

The final step is to import and deploy an example project using nevisAdmin 4:

1. In the nevisAdmin 4 Welcome or Project Settings screen, import the following project: project_CLOUD-PROJECT_20200519T102912Z.zip Adapt the project:
2. Set the latest libraries in the Administration tab under Standard libraries.
3. Open the just imported Cloud-Project project in the Configuration tab. Note that the patterns nevisAuth/nevisProxy Remote Session Store and nevisIDM Database are marked with a red bullet point - this is because their database hostname is incorrect. Correct the database hostname of these patterns.
4. In the Administration > Inventory Settings screen, import the following inventory: inventory_CLOUD-INVENTORY_20200519T102841Z.zip Adapt the inventory:
5. Enter the URL of your GitHub repository.
6. Enter the DOMAIN for the proxy-host-name.
7. Enter the API URL of your Kubernetes cluster using HTTPS and port 443. As long as nevisadmin4 is inside the cluster https://kubernetes.default.svc:443 can be used.
8. Enter the token of your Kubernetes cluster. Get the token by executing the following commands:

Get the token of your Kubernetes cluster

# Find the name of the secret containing the token, this is the token of the service account the operator uses
kubectl get secret -n nevisoperator-system | grep default

# Use the secret name from the output from the command above
kubectl describe secret -n nevisoperator-system default-token-<random_id> | grep token

3. Click the Deploy button and go through the steps of the Deployment Wizard. If there is a warning, you can just accept it and continue.
4. Go to the following URLs to test:

URLs to test

# base url can differ depending on the location

# nevisIDM admin reachable here, default credentials: bootstrap/generated
google-chrome https://$DOMAIN/nevisidm/admin

# Example standalone nevisAuth flow
google-chrome https://$DOMAIN/