Skip to main content

Automatic Key Management

nevisAdmin 4 can automatically generate key material for Nevis components. Automatic key management is a convenience feature which allows you to deploy without concern for keystore and truststore content.

If you have complex requirements, consider the following alternatives:

  • PEM-Based Key Management
  • nevisKeybox-Based Key Management (classic deployment only)

The properties of automatic key management are as follows:

Pros:

  • Keys are created on the target host / inside the Kubernetes cluster and never leaves those systems.
  • Easier to get started as less configuration is required.
  • One shot deployment in case the entire project is deployed.

Cons:

  • Does not support using the same key on multiple hosts / for different services.
  • Limited configuration options.
  • Partial deployment of a project can lead to incomplete truststore content.
  • In classic, not all failure cases (for example deleted files or expired CA) can be fixed in one deployment. Multiple deployments may be required.

To enable automatic key management, first set the following global variable in your inventory.

vars:
  __nevisadmin_pki_automatic_enabled: true

By default, it is not required to add keystore/truststore patterns to your project. The required keystores and truststores will be generated automatically (implicit mode).

As of nevisAdmin version 4.5, you can set the following variables to force users to assign Automatic Key Store / Automatic Trust Store patterns wherever a keystore/truststore is needed:

vars:
  __nevisadmin_pki_automatic_enabled: true
  __nevisadmin_pki_automatic_allowed_mode: explicit

Deployment Scenarios​

Automatic key management works differently in classic VM and Kubernetes deployments. This section highlights the differences.

Kubernetes​

During deployment, nevisAdmin 4 creates NevisKeyStore and NevisTrustStore resources. You can have a look at these resources in the preview step of the deployment wizard. When you continue with the deployment, the resources are sent to the cluster. Here, the nevisOperator component reads and processes them. For each NevisKeyStore resource, the nevisOperator requests a certificate from the used signer. Truststores are populated based on associated keystores. The keystore/truststore files are stored in Kubernetes secrets. The nevisOperator ensures that the secrets are mounted into the correct pods.

You configure the properties to be used for the certificate signing process in the file nevisoperator.yml. The following properties can be configured:

Property NameDefaultDescription
csr-default-values.countryUsed for CSR
csr-default-values.provinceUsed for CSR
csr-default-values.organizationUsed for CSR
csr-default-values.organizational-unitUsed for CSR
csr-default-values.email-addressUsed for CSR
(DEPRECATED) key-management.cluster-signing-ca.sourceSource of the signing CA. Should be used if the cluster is used for signing the certificates, but the signing CA is different than the cluster CA, this can be the case in OpenShift. Accepted value: configmap.
(DEPRECATED) key-management.cluster-signing-ca.configmap.nameName of the configmap containing the CA cert. Key should be cert.pem.
(DEPRECATED) key-management.cluster-signing-ca.configmap.namespaceNamespace of the configmap containing the CA cert.

cert-manager​

cert-manager is used to create the certificates, this requires at least cert-manager v1.0. The following properties have to be configured in the nevisoperator.yml:

Property NameDefaultDescription
key-management.cert-manager.issuer.nameName of the cert-manager issuer to be used.
key-management.cert-manager.issuer.namespaceThe namespace of the issuer. Has to be specified if it is a namespaced Issuer, and not a ClusterIssuer.
key-management.cert-manager.signing-ca-secret.nameName of the secret containing the the CA used by the specified issuer. Secret has to have a key ca.crt, or a single key.
key-management.cert-manager.signing-ca-secret.namespaceNamespace of the secret containing the CA used by the issuer.
key-management.cert-manager.certificate-duration2160hAccepts a Go time duration string specifying the requested certificate duration. Defaults to 90 days.

nevisOperator uses a CertificateRequest to create the certificate with cert-manager.

For upgrading to the latest version of cert-manager and creating an issuer, see Kubernetes Upgrade.

An example configuration that uses the created issuer:

key-management:
  cert-manager:
    issuer:
      name: ca-issuer
    signing-ca-secret:
      name: ca-root-secret
      namespace: cert-manager
    certificate-duration: 8760h

Expiration date​

The expiration date can be configured with the certificate-duration parameter as seen above. nevisOperator automatically renews the certificates one month before expiration - there is no need for additional deployment.

The automatic renewal triggers a rolling deployment, causing the component pods to use the new key materials. You can avoid this by performing a canary deployment at least once a year. This already generates new key materials for the secondary deployment.

Classic​

The deployment process of keystores / truststores in classic VM setups is as follows.

In a classic deployment, only use automatic key management for testing as the generated key material does not work in some deployment scenarios. For production environments, use nevisKeybox-based or PEM-based key management.

The deployment of a keystore on a target host involves several steps:

  1. nevisAdmin 4 runs checks on the target host to validate the integrity of the keystore.
    1. Subsequent commands are only executed if any of these checks fail.
  2. If needed, nevisAdmin 4 runs a command to create a private key and a certificate signing request (CSR) on the target host.
    1. The private key never leaves the target host.
  3. nevisAdmin 4 downloads the CSR and signs it with its own internal nevisAdmin 4 CA.
  4. CSR and certificate are stored in the nevisAdmin 4 DB.
    1. This is required for subsequent deployments and to populate truststores.
  5. The certificate is uploaded to the target host in PEM format.
  6. nevisAdmin 4 runs another command to create a bundle of the private key, certificate and CA chain in various formats (PEM, JKS, PKCS12).

Automatic key management can recover from most failure cases, for example a missing private key or expired certificates; but in classic deployements, it is sometimes required to deploy twice.

Maintenance: Re-creating the nevisAdmin 4 CA​

The nevisAdmin 4 CA has a validity of one year. As there is no automatic renewal process, you have to determine when the nevisAdmin 4 CA will expire and re-create it before that date.

Checking the nevisAdmin 4 CA Expiration Date​

The nevisAdmin 4 CA is added to all automatic truststores. You can therefore check the CA's expiration date in any truststore.pem file in the /var/opt/keys/trust directory. Proceed as follows:

  1. To find the truststore.pem files, run the openssl command shown in the next code block.
  2. Determine the expiration date of the CA.
# find /var/opt/keys/trust -name truststore.pem -exec openssl x509 -in {} -text \; | egrep -B1 "Subject: CN=nevisAdmin 4 CA"
            Not After : Jan 21 14:00:58 2021 GMT
        Subject: CN=nevisAdmin 4 CA, OU=NEVIS Security

Enforcing the Renewal of nevisAdmin 4 CA​

As of nevisAdmin 4.6, there is no GUI or REST API to control the CA. The only way to exchange the CA is via the database.

nevisAdmin 4 will automatically create a new CA when no CA is found in the database. Associated certificates will be re-created with the next deployment. You can therefore just delete the CA in the database. Run the following SQL commands on the nevisAdmin 4 host:

# stop nevisAdmin 4
[root@nevisadmin4 ~]# nevisadmin4 stopService

# connect to the database as configured in /var/opt/nevisadmin4/conf/nevisadmin4.yml
[root@nevisadmin4 ~]# mysql -u <user> -p<password>

MariaDB [(none)]> use nevisadmin4;

# delete the nevsiAdmin 4 CA
MariaDB [nevisadmin4]> delete from ca;

# delete all the stored certificates to enforce that they are created again
MariaDB [nevisadmin4]> delete from pki_store_content;
MariaDB [nevisadmin4]> commit;
MariaDB [nevisadmin4]> quit

# start nevisAdmin 4
[root@nevisadmin4-dev ~]# nevisadmin4 startService

Explicit Usage​

Wherever a keystore / truststore is needed, you can assign the Automatic Key Store / Automatic Trust Store patterns to make the automatic key management explicit.

Further, explicit assignment is required in some patterns to enforce the use of 2-way TLS:

  • Assign an Automatic Key Store to use a client certificate for outbound TLS connections.
  • Assign an Automatic Trust Store to set up a truststore to validate incoming connections.

Linking an Automatic Trust Store with Automatic Key Store(s)​

In the Automatic Trust Store you can assign one or multiple Automatic Key Store patterns via Trusted Key Stores. Use this method to inform nevisAdmin 4 about a trust relation. nevisAdmin 4 will make sure that any certificate issued for that keystore is automatically trusted. This is often required when using generic patterns, for example Generic Authentication Realm or Generic Authentication Step, which perform outbound TLS connections, create or validate tokens.

How to link an automatic truststore with an automatic keystore