Skip to main content

Kubernetes Upgrade

To upgrade the environment running on Kubernetes to the next release, follow these steps. Read all the instructions before you begin the upgrade process.

Upgrade for helm installations​

To upgrade the installation, copy over the new images from Upload Nevis Docker Images then run:

# For the temporary credentials, click the download button for one of the Docker images at https://portal.nevis.net/portal/secure/releases/rolling
CLOUDSMITH_PASSWORD=<cloudsmith-pass>
RELEASE_NAME=<release-name>
RELEASE_NAMESPACE=<release-namespace>

helm upgrade nevisadmin4-crd nevisadmin4-crd --repo https://dl.cloudsmith.io/$CLOUDSMITH_PASSWORD/nevissecurity/rolling/helm/charts/
helm get values $RELEASE_NAME -n $RELEASE_NAMESPACE > old-values.yaml
helm upgrade $RELEASE_NAME nevisadmin4 -n $RELEASE_NAMESPACE -f old-values.yaml \
    --repo https://dl.cloudsmith.io/$CLOUDSMITH_PASSWORD/nevissecurity/rolling/helm/charts/ \
    --version <version-to-upgrade>

If the version is not provided, it will upgrade to the latest one.

Helm changelog​

  • v8.2505.0
    • Updated the used Nevis component versions to the 8.2505.0 release.
  • v8.2411.0
    • Updated the used Nevis component versions to the 8.2411.0 release.
  • v8.2405.0
    • Updated the used Nevis component versions to the 8.2405.0 release.
    • Added configuration values for OpenTelemetry and Product Analytics
    • Gitea repositories will now be initialized when using the bootstrap option
  • v7.2402.1
    • Fixed the usage of the Git credentials when using the git.username and git.password values.
  • v7.2402.0
    • Updated the used Nevis component versions to the 7.2402.0 release.
    • Added values nevisAdmin4.livenessProbe and nevisAdmin4.readinessProbe to be able to define custom liveness and readiness probes.
  • v7.2311.1
    • Added value database.type to support PostgreSQL database.
  • v7.2311.0
    • Switched to new version schema.
    • Updated the used Nevis component versions to the 7.2311.0 release.
    • Added value nevisOperator.image.tag to make it possible to specify a separate version for nevisOperator.
    • Added value nevisAdmin4.extraEnvs to configure additional environment values for nevisAdmin 4.
    • Added value nevisAdmin4.enabled to make it possible to only deploy the nevisOperator component.
    • Added value nevisAdmin4.ingress.annotations to configure additional annotations for the ingress of nevisAdmin 4.
  • v1.4.3
    • Updated the used Nevis component versions to the 4.20.2 release.
  • v1.4.2
    • Fixed an issue that could cause the installation to fail if https authentication was used for git.
  • v1.4.1
    • Fixed an issue that could cause the installation to fail if tls was enabled for nevisAdmin 4.
  • v1.4.0
    • Updated the used Nevis component versions to the 4.20.0 release.
    • Added value nevisOperator.defaultImagePullPolicy to configure the imagePullPolicy of the deployed Nevis components.
    • Secret handling was refactored to use the specified secrets directly, this improves compatibility with tools that depends on helm template.
      • The following values are now deprecated: git.httpCredentialSecret, git.sshCredentialSecret, database.root.credentialSecret. They are still honored but will be removed in the future.
      • caution

        When using the new database.root.preparedCredentialSecret property, the root-creds secret will no longer be created. Adjust the Root Credential and Root Credential Namespace in the database patterns of nevisAdmin 4 to use the prepared secret before the migration to this value.

  • v1.3.2
    • Use standard duration string for the CA certificate
  • v1.3.1
    • Fixed an issue that could cause the upgrade to fail if nginx.controller.ingressClassResource was defined in the provided values.yaml.
    • Fixed an issue that caused the wrong ingress class to be used in nevisOperator if nginx.controller.ingressClassResource.enabled was set to false.
  • v1.3.0
    • Updated the used Nevis component versions to the 4.19.1 release.
  • v1.2.0
    • Updated the used Nevis component versions to the 4.19 release.
    • Added optional mariadb and gitea dependency.
    • Added value repositoryUrlMap to allow configuring separate git repository url for different component namespaces.
    • Added values for cors and ldap.
    • Added values for overriding nevisAdmin 4 configuration files.
    • Fixed an issue that caused an incorrect passphrase generated for the tls keystore.
    • Added support for Kubernetes v1.26, and removed support for v1.21.
  • v1.1.1
    • Fixed an issue that the initial random password of nevisAdmin 4 was generated with each install. This caused the password in the Kubernetes secret and the actual password to be different.
  • v1.1.0
    • Updated the used Nevis component versions to the 4.18 release.
    • Added separate service account for the nevisAdmin 4 instance.
    • Added new label and annotation values.
    • Adjusted used securityContext for restricted Pod Security Standard.
    • Added multiple new values for credentials that can be used in place of prepared secrets.
    • Updated ingress-nginx chart to 4.4.0.
    • Added support for Kubernetes v1.25, and removed support for v1.20.
  • v1.0.1
    • Added cert-manager related values.
    • Fixed null error when using the --dry-run option.

Performing the upgrade using the wizard script​

EXPERIMENTAL

The Kubernetes upgrade wizard script walks you through the Kubernetes upgrade process with minimal user input required. It can also roll back changes, to some extent.

Download the script kubernetes-upgrade.zip.

This script is in an experimental state. For production environments, it is therefore better to perform the Kubernetes upgrade manually. For instructions, see further below.

Arguments & flags​

ArgumentsDescription
-h, --helpPrints usage information.
-r, --run-optionSets what you want to upgrade. Options: all, nevisadmin, operator, images, crd, permissions, cert-manager, nginx; Default: all; For more details on each option, see the table below.
-n, --nevisadmin4-namespaceDefines the namespace of nevisAdmin4. Must be set if --run-options is set to nevisadmin or all.
-o, --operator-namespaceDefines the namespace of nevisOperator. Must be set if --run-options is set to operator or all.
-c, --container-registrySets the container registry to which the new images are uploaded. This is also the registry from where to pull the images for nevisAdmin4 and nevisOperator. Has to be set.
-i, --internal-registrySets the container registry URL from where the images are pulled by the pods.Default: the value of --container-registry.
--rollbackUse this flag to roll back the operation(s) set in --run-options to their previous state, instead of having them upgraded. Rollback limitations: Note that you cannot roll back the following changes once they are executed by the script: Images being copied to the container registry specified in the --container-registry option; Database schema upgrades of nevisAdmin4; Custom resource definition updates; Permission updates; nginx update; cert-manager, in case it was upgraded from 1.4 to 1.7 (upgrades from 0.10 to 1.4 can be rolled back)

Run options​

--run-optionsDescription
imagesCopies the images to the container registry specified in the --container-registry parameter, from the source registry specified in the config.yaml file.
nevisadminUpdates the database schema of nevisAdmin 4, then nevisAdmin 4 itself. Finally, it restarts nevisAdmin 4.
operatorUpgrades nevisOperator.
crdUpdates the custom resource definitions.
permissionsUpdates the permissions.
cert-managerUse this run-option if cert-manager was installed using the manifest provided in this installation guide. It installs the new version of cert-manager. It also adds an additional CA ClusterIssuer called ca-issuer, which can be used for automatic key management. Do not use this run-option If cert-manager was installed separately.
nginxUse this run-option if nginx was installed with the manifest provided in this installation guide. It installs the new version of nginx. Do not use this run-option If nginx was installed separately.
allRuns all of the above, except for cert-manager and nginx.

The following code snippet shows a sample upgrade script, with the default values used by the installation guide:

./upgrade.sh -r all -c <your-registry>.azurecr.io -o nevisoperator-system -n nevisadmin4

Additional configurations​

The script includes the file config.yaml, which contains some additional configuration options.

The following configuration options are noteworthy:

  • source-registry and source-registry-image-prefix: The script will copy images from this repository to the registry specified by the --container-registry argument. The image prefix is used to determine the full path of the images. For example, in case of docker.cloudsmith.io/nevissecurity/rolling/nevisproxy:4.5 the value of source-registry-image-prefix would be nevissecurity/rolling.
  • db-schema-job: Specifies the name of the DB schema migration job. In case you renamed the job, replace the default value with the new name.

Manual Upgrade Instructions​

The following changes were made to the default configuration found in "Installation on supported platforms". We recommend that you apply them to your own setup as well.

Make sure you are using nevisAdmin4 4.14+, cert-manager 1.6+ and ingress-nginx 1.0+ before upgrading to Kubernetes 1.22

Upgrade from nevisAdmin 7.2402.0 to 8.2405.0+​

kubectl apply -f operator.nevis-security.ch_neviscomponents-8.2405.0.yaml

Instructions for previous versions​

Upgrade from nevisAdmin 4.17 to 4.18+​

Upgrade from nevisAdmin 4.18 to 4.19+​

kubectl apply -f nevisadmin4-rbac-4.19.yaml

Alternatively, you can use the wizard script with the permissions run option.

kubectl apply -f operator.nevis-security.ch_neviscomponents-4.19.yaml
kubectl apply -f operator.nevis-security.ch_nevisdatabases-4.19.yaml

Upgrade from nevisAdmin 4.19 to 4.20+​

kubectl apply -f operator.nevis-security.ch_neviscomponents-4.20.yaml
kubectl apply -f operator.nevis-security.ch_nevisdatabases-4.20.yaml
kubectl apply -f operator.nevis-security.ch_nevisingresses-4.20.yaml

Upgrade from nevisAdmin 4.20 to 7.2311.0+​

Upgrade from nevisAdmin 7.2311.0 to 7.2402.0+​

kubectl apply -f operator.nevis-security.ch_neviscomponents-7.2402.0.yaml
kubectl apply -f operator.nevis-security.ch_nevisingresses-7.2402.0.yaml
kubectl apply -f operator.nevis-security.ch_nevisdatabases-7.2311.0.yaml

Uploading New Docker Images​

Upload the Docker images for the current release as described in Uploading Nevis Docker Images of the Example Installation on Azure chapter.

Upgrading nevisAdmin 4​

  1. (Optional) You can use the kubectl cp command to back up the files described in the Database and File System Backup chapter.

    # Find out which pod is used for nevisadmin
    kubectl get pods --all-namespaces | grep nevisadmin4-

    # Example for getting the configuration files.
    # Update the pod name and namespace and copy the configuration files to the local machine
    kubectl cp <namespace>/<podname>:/var/opt/nevisadmin4/conf conf

  2. Upgrade the nevisAdmin4 database: Copy the migration job called nevisadmin4-dbschema from kubernetes-guide or build/nevis-cluster/kubernetes-nevisadmin4/nevisadmin4.yamlto a separate file, then update the image version. Check the following example:

    apiVersion: batch/v1
    kind: Job
    metadata:
     name: nevisadmin4-dbschema
     namespace: nevisadmin4
    spec:
     activeDeadlineSeconds: 300
     template:
     spec:
     restartPolicy: Never
     containers:
     - name: nevisadmin4-dbschema
     image: nevisguidetest.azurecr.io/nevis/nevisadmin4-dbschema:<CURRENT-VERSION>
     imagePullPolicy: Always
     env:
     - name: NEVIS_DBSCHEMA_URL
     value: nevisguidetest.mariadb.database.azure.com:3306/nevisadmin4
     - name: NEVIS_DBSCHEMA_HOSTNAME
     value: nevisguidetest.mariadb.database.azure.com
     - name: NEVIS_DBROOT_USER
     value: nevisguidetest
     - name: NEVIS_DBROOT_PASSWORD
     value: gdfgdf434FVUHGVHGddsd---
     - name: NEVIS_DBSCHEMA_USER
     value: nevisguidetestuser
     - name: NEVIS_DBSCHEMA_PASSWORD
     value: nevisguidetestuser
     - name: NEVIS_DBUSER_USER
     value: nevisguidetestschema
     - name: NEVIS_DBUSER_PASSWORD
     value: nevisguidetestschema
     command:
     - "nevis-dbschema"
     - "migrate"
     resources:
     requests:
     cpu: 20m
     memory: 200Mi
     limits:
     cpu: 1000m
     memory: 1000Mi

    If the old files are no longer available, you can get the configuration used by looking at the old job:

    # namespace is where nevisadmin4 instance is running, by default it is nevisadmin4
    kubectl get job nevisadmin4-dbschema -n <namespace> -o yaml

    Recreate the job:

    # delete old job, by default the namespace is nevisadmin4
    kubectl delete job nevisadmin4-dbschema -n <namespace>

    # create new one, with the previously updated file
    kubectl apply -f <filename> -n <namespace>

  3. Upgrade the running nevisAdmin4 instance. Change the image versions used by the current deployment:

    # image version is given with the registry, example: nevisguidetest.azurecr.io/nevis/nevisadmin4-dbschema:4.8
    # namespace is where nevisadmin4 instance is running, by default it is nevisadmin4
    # full example: kubectl set image statefulset/nevisadmin4 nevisadmin4=nevisguidetest.azurecr.io/nevis/nevisadmin4:4.8 nevisadmin4-dbschema-wait=nevisguidetest.azurecr.io/nevis/nevisadmin4-dbschema:4.8 -n nevisadmin4

    kubectl set image statefulset/nevisadmin4 nevisadmin4=<nevisadmin4-registry-and-version> nevisadmin4-dbschema-wait=<nevisadmin4-dbschema-registry-and-version> -n <namespace>

Upgrading nevisOperator​

Change the image versions used by the current deployment:

# image version is given with the registry, example: nevisguidetest.azurecr.io/nevis/nevisoperator:4.8
# namespace is where the operator instance is running, by default it is nevisoperator-system
# full example: kubectl set image deployment/nevisoperator-controller-manager manager=nevisguidetest.azurecr.io/nevis/nevisoperator:4.8 -n nevisoperator-system

kubectl set image deployment/nevisoperator-controller-manager manager=<registry-and-version> -n <namespace>

Upgrading the operator may cause a rolling deployment to start if the new operator version causes a change in the deployment definitions. See also the Release Notes.

If your environment is not working smoothly after the upgrade, check the chapter Kubernetes Deployment Troubleshooting for debugging instructions.

Downtime during upgrade​

From 4.12 onward, PodDisruptionBudgets are automatically created for the deployments, see Kubernetes Infrastructure Inventory YAML file format among the template files. This means that if at least two replicas are used, at least one replica is constantly available during the upgrade process. It is recommended to also create a PDB for the nginx controller, as without that there may be some service downtime. See an example for this in the nginx.yaml, which is provided in Installation among the template files.

Migrating to cert-manager​

Using the cluster as the signer is deprecated as of 4.13, and will be removed in a future release.

From now on, cert-manager is the recommended way to generate certificates when using automatic key management. For more information, see cert-manager. For this purpose, cert-manager 1.0+ is required. If you already use such a version, the upgrade process can be skipped, and only the new Issuers need to be applied and the configuration for nevisOperator modified.

Upgrading cert-manager​

If cert-manager was installed separately, follow the official guide.

If cert-manager was installed with the manifest provided in the installation guide, do the following:

(Optional) Create a backup for the existing issuers, this is only necessary if you created additional Issuers manually:

kubectl get --all-namespaces -o yaml issuer,clusterissuer,cert > backup.yaml

Remove the current installation with the cert-manager-remove.yaml manifest:

kubectl delete -f cert-manager-remove.yaml

Install the new version of cert-manager with using the official manifest

kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.7.0/cert-manager.yaml

Adding the Issuers​

Add issuers with issuers.yaml, which also creates an additional CA ClusterIssuer called ca-issuer for automatic key management:

kubectl apply -f issuers.yaml

Extending nevisOperator configuration​

First make sure you applied the newest RBAC file for nevisOperator, this is automatically done by the upgrade script.

To use the issuers created above, add the following to the nevisOperator config, and then restart nevisOperator. It is also possible to create your own Issuer and refer to that one. For more information about the used parameters see: cert-manager

key-management:
 cert-manager:
 issuer:
 name: ca-issuer
 signing-ca-secret:
 name: ca-root-secret
 namespace: cert-manager
 certificate-duration: 8760h

Either add it to the file in kubernetes-guide/src/nevis-cluster/kubernetes-operator/operator-config.yaml and apply it after generating the configuration with the gradle task, or edit the existing configmap:

kubectl edit configmap nevisoperator-operator-config-6fc9hfhdhk -n nevisoperator-system

You can restart nevisOperator with:

kubectl rollout restart deployment nevisoperator-controller-manager -n nevisoperator-system

Upon restart nevisOperator will replace all the key material used by the current deployments, this can cause a short downtime, while the components are restarting with the new key material.