nevisKeybox
Welcome to the nevisKeybox documentation.
The nevisKeybox release package is available for the following operating systems:
- Red Hat Enterprise Linux x86-64, version 6 or 7
- SUSE Linux Enterprise Server x86-64, version 11 or 12
- nevisAppliance v2 (pre-installed)
Concept Description
What is it about
The terms nevisKeybox and Keybox are used interchangeably.
nevisKeybox is a simple PKI administration utility required to perform administration tasks on hosts, which need to generate certificate requests and import these again when returned from the CA.
The nevisKeybox is a local component that may be installed on any host hosting a Nevis component or site-specific service that requires PKI. It therefore does not need to be integrated with any other Nevis components, but provides a service for these. The following overview provides an idea, where certificates are used to configure a Nevis environment. Note that 2-way SSL connections are used everywhere and the Nevis identity propagation mechanism is in place.
The following certificates are used:
- Client Certificate Authentication (user certificates from user CAs)
- External/public communication (node certificates usually from public CAs like Verisign or SwissSign)
- Internal/private (VPN) communication (node certificates from internal company CAs)
- Nevis security token signer (locally generated, self-signed certificates or from internal special purpose CAs)
Main features
The nevisKeybox utility supports:
- A keytool-oriented command line interface
- A file-based certificate and key store (using OpenSSL and Java)
- An HSM certificate and key store (using PKCS#11-enabled OpenSSL)
- A certificate management reduced to the max by automatically providing different certificate and key formats as well as a simple policy configuration.
- Providing a well known policy to store key material in the machine’s file system (for access by different application and processes), including file location, file access and passphrase protection.
- Providing a simplified usage pattern to OpenSSL and keytool utilities by providing defaults, validating input, and giving workflow hints.
Ad hoc certificate signing with an ad hoc root- or sub-CA is part of the nevisKeybox (by just using the OpenSSL CA module), but this feature is only recommended for development purposes or if no CA infrastructure is available.
Terminology & Naming conventions
Most of the terms below are common when dealing with PKI components.
Term | Explanation | Remarks |
---|---|---|
keystore | A keystore usually contains an X509 user or node certificate along with its private key. Usual purpose: node or user authentication. | |
truststore | A truststore contains X509 certificates (usually CAs). Usual purpose: peer verification, signature verification. | |
slot | A slot is a compartment to store key material. | HSM: Slot or Partition. Java: A keystore file. |
label | Used to identify key material with a logical name. | HSM: Label. Java: Alias. |
Proposal for slot names
Slots should relate to network tiers or physical software components to reflect the area in which a certificate object is used.
Naming Convention | Explanation | Detailed names |
---|---|---|
default | Area inside or behind the reverse proxy (usually a VPN). | intranet, authentication zone, VPN |
public | Area in front of the nevisProxy, usually certificates signed by an external CA like Verisign. | internet, extranet |
nevis | Key material, related to the Nevis identity passing mechanism, containing the SecToken signer key- and / or trust-store (label auth recommended). |
Proposal for label names
Label names define what a key object is used for. The following table shows the default usage names suggested to use when working with Nevis components.
Naming convention | Explanation | Detailed names |
---|---|---|
node | This is referring to a private cert object used for node authentication on SSL/TLS network connections. | The node’s DNS name. |
File-system-based keybox
Managing certificates in the local file system often leads to these questions:
- How do I generate certificate requests?
- How shall I protect the otherwise insecure key material (ownership, permissions, passphrases)?
- How do I locate key material, used by locally installed components?
- How do I convert one format (e.g. PEM) into another (e.g. JKS)?
The nevisKeybox handles these questions as follows:
- nevisKeybox runs as user root. All sensitive key material generated has read/write permissions for root only (this is sufficient for services like proxies usually binding port 443 and therefore requiring root access anyway). Access may be granted for one OS group only (not world). Access for multiple groups requires cloning.
- nevisKeybox supports generated passphrases (unknown to the administrator) for automatic service startup. The password getter has the same access restrictions as the sensitive key material.
- nevisKeybox automatically provides key material in all commonly required formats (PEM, DER, PKCS8, JKS, PKCS12).
- The file-system-based nevisKeybox provides default policies for storage location and file names. A file path is constructed as follows:
/var/opt/neviskeybox/<instance>/<slot>/<label>_<contenttype>.<format>
where:
<instance>
is the nevisKeybox instance name.<slot>
is the compartment to store logically related key material.<label>
is a logical name to identify its content or usage.<contenttype>
specifies the type of the data in the file.<format>
is the encoding of the file content.
The truststore policy is:
/var/opt/neviskeybox/<instance>/<slot>/truststore.<format>
- The HSM-based keybox treats certificates within an HSM slot having a certificate, public and private key, as keystores and all the other certificates as one single truststore. File-system- and HSM-based keystores (including JKS) therefore share the same data model.
- nevisKeybox suggests the following default policies: 2048 bit RSA keys with 3DES protection 720 days certificate request validity.
Formats
Key material is provided in all of the following formats (whereas the administration module provides all the required conversions):
Format | Consumer | Usage | Remarks |
---|---|---|---|
JKS | Java-based application. (JBoss, WildFly, Jetty, …) | Keystore, truststore | A keystore only contains one private key, protected by the same password as the keystore itself. |
PKCS12 | Java-based application, Browser import. | Keystore | Truststore with OpenSSL not possible without private key. Alternate format to JKS. |
PEM (X509) | C/C++/OpenSSL-based application (nevisProxy, OpenLDAP, …) | Keystore, truststore | Keystore contains an encrypted private key. |
DER (X509) | nevisAuth Java-based application. | Certificate | Oracle Java JCE does not support encrypted private keys. |
DER (PKCS8) | nevisAuth Java-based application. | Private key | Oracle Java JCE does not support encrypted private keys. |
HSM-based keybox
The HSM subsystem of the nevisKeybox provides the same administration API to manage PKCS#11-based Hardware Security Modules (HSM). As keys and certificates are not stored in the file system(3), file permission problems are not an issue here. Access is usually permitted if the caller knows the passphrase to access the HSM slot in question.
To access HSMs, the following preconditions need to be met:
- The vendors PKCS#11 driver library has to be installed on the system
- The HSM slot needs to be mapped into the nevisKeybox
- The PKCS#11-enabled OpenSSL executable delivered with the nevisKeybox package (containing the pkcs11 application module) needs to be used.
Supported HSM
Currently nevisKeybox supports the following HSM vendors:
- Gemalto LunaSA
Installation
This chapter describes the steps that perform to install nevisKeybox. The package is operating-system-specific and uses the package mechanism of the target OS.
Available Packages
The following packages are available:
- neviskeybox: RPM package for SUSE Linux Enteprise Server (SLES) 11 and 12 and Red Hat Enterprise Linux (RHEL) 6 and 7.
Prerequisites
The following requirements need to be fulfilled before the nevisKeybox package can be installed:
- SLES 11, 12 or RHEL 6, 7 Linux host with about 20MB free disk space.
- OpenJDK 1.8.0 package by the operating system or adnjdk18 package.
- Perl 5+, Perl::DataDumper module, file and which installed on the host.
The following command installs all required dependencies for RedHat 7:
yum install \
java-1.8.0-openjdk-headless \
file \
which \
perl \
perl-Data-Dumper
Installing nevisKeybox
Proceed as follows to install the package on Linux:
$> rpm -i neviskeybox-2.x.x.0-1.noarch.rpm
It is recommended to create the default instance after installation. If the directory /var/opt/neviskeybox/default does not yet exist, run the following after installing the RPM:
$> neviskeybox handover
Deinstalling nevisKeybox
Proceed as follows to remove the nevisKeybox software (without deleting log files and configuration):
$> rpm -e neviskeybox
The existing nevisKeybox instances and the key material are not removed along with the package. It is possible to use them again when a new software version is installed.
Software Upgrade
Proceed as follows to upgrade to a new software version without changing the configuration:
- (optional) Deinstall the old package (see: Deinstalling nevisKeybox).
- Install the new package (see: Installing nevisKeybox).
Operation
This chapter describes all aspects of operating and managing key material with nevisKeybox, including the command line interfaces.
Readers of this chapter should be familiar with UNIX command-line interfaces. There are two commands:
neviskeybox: Unix wrapper to manage multiple nevisKeybox setups.
keybox: Manage a single (default) keybox instance.
Fading outIt's recommended to use the neviskeybox command instead, to avoid any side-effects and have instance handling, see: "nevisKeybox Command", for more details.
Operation - Components
This chapter describes the components used for the operation of nevisKeybox.
nevisKeybox Command
After installing the nevisKeybox package, it provides the neviskeybox command-line wrapper. This wrapper is used to:
- Create and manage multiple keybox setups (instances).
- Assert a correct runtime environment, using sudo (a facility to provide a secure interface to authorize UNIX users and enforce a well-defined environment in which commands are executed).
- Install a link in /usr/bin, so that the user’s execution environment (PATH) does not have to be extended.
The nevisKeybox default instance is created in the post-installation procedure of the package and is located at /var/opt/neviskeybox/default. A legacy link /var/spool/keybox is created, which points to this location (for backward compatibility and shorter keybox configuration references).
The command is available at the following locations:
- /usr/bin/neviskeybox
- /opt/neviskeybox/bin/neviskeybox
- /opt/neviskeybox/2.x.y.z/bin/neviskeybox
If you prepend the package location /opt/neviskeybox/bin to your PATH or use /opt/neviskeybox/bin/keybox directly, the script keybox is executed without using the command wrapper neviskeybox, which in turn provides instance support and environment setup. To prevent sideeffects, use the neviskeybox command.
keybox Convenience Command
To avoid side-effects and have instance support, it's recommended to use the neviskeybox command instead, see above.
The keybox command-line executable can also be used directly, i.e. by executing:
$> export PATH=/opt/neviskeybox/bin:$PATH
$> keybox
The command checks the following preconditions:
- The executing user must be identical to the owner of the keybox files.
- If no keybox.conf file is found at the current location, the command tries to write the changes to /var/spool/keybox. This is a link to the default neviskeybox instance located at /var/opt/neviskeybox/default.
If your installation provides an administration wrapper such as sudo, however, it is recommended to integrate the keybox command-line executable into one of these facilities.
The command is available at the following locations:
- /opt/neviskeybox/bin/keybox
- /opt/neviskeybox/2.x.y.z/bin/keybox
OpenSSL
Keybox uses OpenSSL for most of the certificate and key manipulations. For further information, refer to the OpenSSL online documentation. OpenSSL may be used from /opt/neviskeybox/bin/openssl
for advanced tasks, e.g.:
- Using s_client to test an SSL/TLS-enabled server
- Display technical details about keystore objects
The openssl executable is delivered with the neviskeybox package.
Java (JRE)
nevisKeybox uses the Java keytool and Java executable if JKS format support is enabled (default).
It is recommended to use the OpenJDK 1.8 JRE. The JRE must be preinstalled. The nevisKeybox package is usually able to locate the OS-specific JRE provided by the OS vendor or the adnjdk package.
Operation - Administration Commands
This command line interface provides four functional blocks.
Basic Commands
- Creating certificate requests for external certificate authorities and associating the returned certificate with the private key (generated during request creation).
- Checking keybox content for expiring certificates.
- Managing file system access and passphrases (file-system-based store only, passphrase getter may be set for HSM slots).
Advanced Commands
- CA functions
- Replication of key material in case of file system permission problems, when multiple processes are using the same files and do not run under a common user (root) or group.
Instance Commands
The commands provided are:
- Listing instances: neviskeybox inst
- Creating the default instance: neviskeybox handover
- Creating instances: neviskeybox inst create myinst
- Removing instances: neviskeybox inst remove myinst
- Addressing instances: neviskeybox myinst ...
When addressing instances, the instance default is used, if the first argument does not address an existing nevisKyebox instance setup.
ZMS Commands
- Synchronize keybox certificate store with the Nevis PKI.
neviskeybox access
This command is required in case of a process that requires access to a key object, but does not run as an OS user root. Access is based on Linux octal file system permissions, see man chmod for details.
There are two possible policies:
- group policy; and
- user policy.
Group policy
A group policy, where the permission for files, containing keys is set from 0600 to 0640 and the group setting is set to the group specified (all processes, requiring access to those keys, need to be a member of this group):
$> neviskeybox access -group bin -slot default -label node
User Policy
A user policy, where the permission for files, containing keys is set from 0600 to 0644 and the keys therefore are only protected by passphrases:
$> neviskeybox access -user bin -slot default -label node
A user policy, where the specified user gets his own keystore copy with an unencrypted private key (PEM only):
$> neviskeybox access -user bin -nopass -slot default -label node
Alternatives
When the user policy is not feasible and the group policy cannot be used, cloning the key could be an alternative:
$> neviskeybox clone -slot default -label node \
-newslot default1 -newlabel node
$> neviskeybox access -group other \
-slot default1 -label node
Cached passphrases (using neviskeybox passwd -keep) are replicated per (nonroot) user with permission 0700, when executing this command. The keybox passphrase getter /opt/keybox/bin/keyboxpwget in turn detects these user specific passphrase getters for non-interactive process startup.
neviskeybox activateCDP
Integration and support for the Nevis certificate management system (ZMS) is being removed in future versions of nevisKeybox.
Performs download and processing of the CDP in one step.
$> neviskeybox activateCDP
neviskeybox cacreate
To create a local CA, the following command may be used:
$> neviskeybox cacreate -ca MyLocalCA
neviskeybox certreq
This command is used to generate a certificate request to be sent to a CA for signing. A private key is generated along with the request (except in the case of a certificate renewal). The generated key is protected by a user-defined passphrase (or a generated one, if specified).
The distinguished name is specified with the -subject option either in the RFC4514 notation (using commas as field separators) or in the OpenSSL notation (using / as field separators).
Supported fields are: UID, CN, OU, O, STREET, L, ST, C, DC.
The emailAddress is usually attached to the CN with a /
.
If the -cn options is used instead, the value is used for the CN field and all other fields are derived from the configuration (see "Keybox, Key and CSR Attributes").
Simple usage (to generate the default node certificate):
$> neviskeybox certreq
Generate an another certificate request (recommended to always specify slot/label options):
$> neviskeybox certreq -slot public -label virtualhost1
Custom usage (customized key size and validity):
neviskeybox certreq -keysize 1024 -days 365
Specify requested certificate type by setting different key usage flags:
- node (default): ssl server/client certificate
- user: ssl client, email signing and encryption
- enc: encryption
- sign: signing
$> neviskeybox certreq -certtype user
Let the keybox generate a random passphrase and passphrase getter for non-interactive process start-up:
$> neviskeybox certreq -genpw
Generate a certificate request without user interaction:
$> # note: instead of setting PASSPHRASE you can also use -genpw
$> PASSPHRASE=some-secure-and-long-passphrase \
neviskeybox certreq -slot public -label node -batch \
-subject 'CN=host.company.com,OU=prod,O=company,C=ch'
Requesting a new certificate for an existing key (certificate renewing, existing private key is reused):
$> neviskeybox certreq -renew
Requesting a new certificate, replacing existing key and request (redo the certreq step):
$> neviskeybox certreq -replace
The following objects exist in the file system after executing this command:
- A PEM-encoded certificate request
- A private key (PEM)
- Converted keys (DER and/or PKCS8)
- If -genpw was specified, a passphrase getter, which contains the random passphrase
A sample CSR generation may look as follows:
$> neviskeybox certreq -subject 'CN=host.company.com,OU=prod,O=company,C=ch'
INFO: slot 'default' does not have a passphrase yet.
Enter new passphrase: *****
Confirm new passphrase: *****
Used CSR subject: /C=ch/O=company/OU=prod/CN=host.company.com
Continue with this subject (enter 'i' to go into interactive mode) ? (y/n/i) y
INFO: Generating key and request for default/node with subject
/C=ch/O=company/OU=prod/CN=host.company.com...
INFO: Generate a new 2048 bit RSA key...
Generating RSA private key, 2048 bit long modulus
.........+++
.................................................+++
e is 65537 (0x10001)
NOTE: Generated key is inactive and will be activated on import.
INFO: Generating CSR...
INFO: Successfully generated CSR
neviskeybox check
This command searches the whole keybox instance for certificates, which will expire within the specified period (default 30 days).
$> neviskeybox check
A possible output may look as follows:
$> neviskeybox check
WARNING: The following certificate will expire within 30 days:
Slot/Label : eracom_openssl / testkey5
Subject : /C=ch/O=adnovum/CN=momo.adnovum.ch
Issuer : /C=ch/O=adnovum/CN=momo.adnovum.ch
Fingerprint : 86:96:86:86:9C:78:8D:06:8C:05:9D:A7:88:B5:B9:75
Validity : Sep 13 14:33:12 2016 GMT - Sep 3 14:33:12 2017 GMT
neviskeybox clone
This command is used to create a copy of an existing keybox object.
$> keybox clone -slot some -label node \
-newslot other -newlabel node
See the neviskeybox access command for further details.
neviskeybox delete
This command is used to delete an existing keybox object:
$> neviskeybox delete -slot some -label other
neviskeybox downloadCDP
Integration and support for the Nevis certificate management system (ZMS) is being removed in future versions of nevisKeybox.
This command downloads the certificate distribution package (CDP) from the ZMS system, verifies its signature, and caches it in the keybox (see CDP directory).
If the signature check fails, the ZMS truststore (see zms slot) does not match the ZMS signers private key or the data has been corrupted.
The common use of this command is:
$> neviskeybox downloadCDP
neviskeybox exportkey
This command is required to get an unprotected key object (as there are components that cannot handle encrypted private keys). The exported key is read-only for the owner specified on the export command:
$> neviskeybox export -file /var/component/certs/privkey.pem -owner bin
See also the neviskeybox access -nopass]" command for a manged alternative to this command. Managed means, that the exported keystore remains in the slot and is regenarated, when the object is changed e.g. due to a certificate renewal.
neviskeybox extractCDP
Integration and support for the Nevis certificate management system (ZMS) is being removed in future versions of nevisKeybox.
This command is used to upload the content of the keybox into a new ZMS system. For this command to work correctly, the following preconditions must be met:
The machine needs to be known in the ZMS system. By default, the fully qualified hostname, returned by the following command, is used:
getent hosts 'uname -n'
This value may be customized by setting the ServerName attribute in keybox.conf.
The ZMS import must be enabled (see below)
The trust anchors within the keybox must be configured in the ZMS database (for the domain or this machine).
As the upload is not authenticated, the upload scenario was designed as follows:
- Host administrator calls ZMS administrator by phone and identifies himself
- Host administrator tells which machine he wants to integrate
- ZMS administrator opens access for this machine (for a limited amount of time, e.g. 2 minutes)
- Host administrator executes the upload (which automatically disables access afterwards)
- ZMS administrator verifies the upload
The common use of this command is:
$> neviskeybox extractCDP
The command extracts all slots from the keybox installation. To ignore locally managed slots, use the following:
$> neviskeybox extractCDP -ignore slot1,slot2
neviskeybox genkey
This command supports generation of a single RSA key with an optional attachment of random bytes intended to be used for symmetrical crypto. The RSA key is used to protect the slot (passphrase handling).
Command usage is as follows:
$> neviskeybox genkey -slot genkey -label symkey -randsize 55
The random key is stored in plain format in the file genkey/symkey_key.bin while the AES256 encrypted form (using the slot passphrase) is appended to the key file genkey/symkey_key.pem as follows:
-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----
-----BEGIN NEVISKEYBOX KEY ATTACHEMENT-----
U2FsdGVkX18L2C+rhOm8SumSfWCyO1Yj+lRpPB2h76+COfLNTfXwl4PrYio+JcD7
vdHY0NzbFCZnWWo7+FHYZ+2LyzqnwFmqUSsiSao4AXU
-----END NEVISKEYBOX KEY ATTACHEMENT-----
The fact that the attachment is encrypted allows the file to be transferred to another location for import into another keybox.
Optional key attachment is not supported for ZMS and HSM integration.
neviskeybox help
This command displays different help and usage screens. To see all options and notes, execute:
$> neviskeybox help
$> neviskeybox help all
To see only options for a specific command:
$> neviskeybox help <command>
# For example:
$> neviskeybox help certreq
neviskeybox import
This command imports a certificate bundle, returned from the CA. The bundle is required to be in PEM format and may contain the following:
- X509 certificate that was requested (mandatory)
- CA certificate chain that was used to sign the requested certificate
- A private key
The import performs the following actions:
- The matching slot/label is detected
- If a certificate request and key exist locally in the keybox, the private key in the bundle is ignored.
- A keystore is generated, containing: X509 user/node certificate and private key (PEM-encoded keystore, PEM- and DER-encoded certificate file, PKCS12 and JKS keystore).
- A truststore is generated containing all X509 CA certificates (PEM-encoded truststore, directory with DER-encoded certificates, JKS truststore).
Import certificate and CA bundle separately:
$> neviskeybox import -file /tmp/new_cert.pem
$> neviskeybox import -file /tmp/ca_chain.pem
Import a bundle containing all certificates:
$> neviskeybox import -file /tmp/ca_bundle.pem
To use node/user certificates as trust material:
$> neviskeybox import -trust -file /tmp/new_cert.pem
A sample import (of a requested certificate and the CA in one single PEM file) may look as follows:
$> keybox import -file /tmp/new_cert.pem
INFO: Reading import file /tmp/new_cert.pem...
INFO: Reading keybox slot default...
INFO: Assigning label 'node1' to trusted certificate
>>> add trusted certificate:
Slot/Label : default / node1
Subject : /C=ch/O=company/OU=prod/CN=Developer CA: myCA
Issuer : /C=ch/O=company/OU=prod/CN=Developer CA: myCA
Fingerprint : 2F:8B:DF:34:5F:AF:6D:D6:75:0D:B7:52:24:AF:55:C8
Validity : Feb 17 15:06:50 2005 GMT - Jul 5 15:06:50 2032 GMT
>>> add own certificate:
Slot/Label : default / node
Subject : /C=ch/O=company/OU=prod/CN=host.company.com
Issuer : /C=ch/O=company/OU=prod/CN=Developer CA: myCA
Fingerprint : 30:82:1D:DA:16:09:AD:46:5F:B5:EA:B4:46:32:FA:F6
Validity : Feb 17 15:07:06 2005 GMT - Jul 5 15:07:06 2032 GMT
NOTE: Found 2 new certificates in import
NOTE: Found 0 private key in import
NOTE: Ignoring 0 objects from import
NOTE: Keeping 0 own certificates in keybox
NOTE: Keeping 0 trusted certificates in keybox
Proceed ? (y/n) y
Enter passphrase for slot 'default':
INFO: Import CA certificate 'node1' into truststore of slot 'default'
Certificate was added to keystore
INFO: Import node certificate 'node' into keystore 'default' (adding).
INFO: Activating RSA key (pem)
INFO: Activating RSA key (der)
writing RSA key
INFO: Activating RSA key (pkcs8)
INFO: Import node certificate 'node' into keystore 'default' (pem)
INFO: Import node certificate 'node' into keystore 'default' (der).
INFO: Import node certificate 'node' into keystore 'default' (jks)
INFO: Import node certificate 'node' into keystore 'default' (pkcs12)
INFO: Import completed.
As the import from a PEM bundle with multiple certificates in it is under specified import of trusted certificates, the label option on the command line is enhanced by a counter. To avoid this, import trusted certificates one by one by specifying your own labels.
neviskeybox list
This command displays slots all the objects in the keybox. The following shows a keybox listing after a certreq and import action on the default slot and default label, i.e., there is one node certificate in all supported formats available:
$> neviskeybox list
###################### FILE BASED SLOTS #######################
NevisBoxCA (adhoc CA for simple usage)
client (regular FS module, locally managed)
default (regular FS module, locally managed)
nevis (regular FS module, locally managed)
public (regular FS module, locally managed)
################# KEYBOX-MANAGED PKCS11 SLOTS #################
INFO: No keybox-managed PKCS11 slots found
###############################################################
List one slot: keybox list -slot
List all slots: keybox list -all
$> keybox list -slot default
___________________________ default ___________________________
Keystore URL: /var/opt/neviskeybox/default/default/node_keystore.pem
Keystore URL: /var/opt/neviskeybox/default/default/node_keystore.jks
Keystore URL: /var/opt/neviskeybox/default/default/node_keystore.p12
Slot/Label : default / node
Subject : /C=ch/O=Adnovum Informatik AG/CN=adnwsrl06.zh.adnovum.ch
Issuer : /C=ch/O=Adnovum Informatik AG/CN=NevisBoxCA (svnode)
Fingerprint : 6B:D8:D6:39:FA:86:96:54:95:FC:3D:DF:C9:59:41:72 (MD5)
Validity : Dec 21 11:58:46 2006 GMT - May 8 11:58:46 2034 GMT
File Access : bin WORLD!!!
Pass Access : root bin nobody
Truststore URL: /var/opt/neviskeybox/default/default/truststore
Truststore URL: /var/opt/neviskeybox/default/default/truststore.pem
Truststore URL: /var/opt/neviskeybox/default/default/truststore.jks
Slot/Label : default / node1
Subject : /C=ch/O=Adnovum Informatik AG/CN=NevisBoxCA (svnode)
Issuer : /C=ch/O=Adnovum Informatik AG/CN=NevisBoxCA (svnode)
Fingerprint : 49:38:4B:07:95:80:12:9B:03:C1:81:57:8B:4E:D2:DF (MD5)
Validity : Dec 21 11:58:37 2006 GMT - May 8 11:58:37 2034 GMT
To see all the slots, execute:
$> neviskeybox list -all
There are other files in the keybox, not displayed above, which may be required for special cases. Use the following command to locate them:
$> find /var/opt/neviskeybox/default
neviskeybox passwd
This changes the protection passphrase for a key object:
$> neviskeybox passwd -slot default -label node
To generate a passphrase getter to avoid being forced to type it on server start-up, the following command may be used:
$> neviskeybox passwd -keep -slot default -label node
The generated passphrase getter in this case (using default slot and label names) is:
/var/opt/neviskeybox/default/default/node_keypass
As an HSM abstraction usually requires a login to a slot and allows access to all key material in that slot, it is recommended to apply the same passphrase to all key material in the same slot. This is enforced with neviskeybox release 2.0 or higher.
neviskeybox processCDP
Integration and support for the Nevis certificate management system (ZMS) is being removed in future versions of nevisKeybox.
This command uses the current certificate distribution package (CDP) from CDP directory, verifies again its signature, and compares its content against the content of the keybox.
The following rules apply:
- Certificates within a ZMS-managed keybox slot, which are not in the CDP, are scheduled for deletion. A ZMS-managed slot is a slot that was referenced once by a CDP download.
- Unchanged certificates are not touched. The certificate fingerprint is checked to detect this.
- Own certificates with a differing fingerprint are replaced if there is a pending certificate request and a new key This happens when an expiring certificate was rekeyed.
- Own certificates with a differing fingerprint but no pending certificate request are ignored (which means, the request may have been lost).
- Trusted certificates with a differing fingerprint are replaced. This happens, when an expiring CA certificate was replaced in the ZMS system.
- Managed slots that are not referenced anymore in the CDP may be set to unmanaged (detaching them from the ZMS) or may be deleted.
The common use of this command is:
$> neviskeybox processCDP
neviskeybox rename
This command is used to change the label of an existing keybox object (in case there have been mistakes when generating the key material):
$> neviskeybox rename -slot some -label other
neviskeybox renewExpiring
This command calls the expiration check function of the keybox and, in addition, starts an interaction with the administrator by asking to automatically re-key an expiring certificate. If the administrator confirms, a new key pair and request is generated. The generated CSR must be uploaded manually to the CA.
The usage to check for an expiry within 30 days is:
$> neviskeybox renewExpiring -expiry 30
neviskeybox revoke/gencrl
Certificates that are signed by a nevisKeybox CA may be revoked as follows:
$> neviskeybox revoke -ca MyLocalCA \
-file /var/opt/neviskeybox/default/enc/my_certificate.pem
As a result, a file CRL can be generated using the following command:
$> neviskeybox gencrl -ca MyLocalCA
The resulting CRL is provided as a PEM-encoded CRL and as an LDIF file that can be imported into an LDAP directory.
neviskeybox selfcert
This command is used to create a self-signed certificate, e.g., used for data signing:
$> neviskeybox selfcert -slot nevis -label auth
The associated truststore contains the certificate itself. For signature verification, the generated certificate (public key) should be propagated to a certificate distribution point (e.g. an LDAP directory).
neviskeybox sign
The keybox contains a small CA that can be used to sign certificates for development and test purposes. When this command is executed for the first time, a CA is generated.
$> neviskeybox sign -file /var/opt/neviskeybox/default/default/node_request.pem
Certificate renewal (i.e. signing a certificate with a subject line that has already been used) is not supported by this CA.
You may specify the key usage flags of the certificate that is issued by setting a template:
- node (default): ssl server/client
- user: ssl client, email signing and encryption
- enc: encryption
- sign: signing
$> neviskeybox sign -file /var/opt/neviskeybox/default/enc/my_request.pem -certtype enc
neviskeybox uploadCSR
Integration and support for the Nevis certificate management system (ZMS) is being removed in future versions of nevisKeybox.
This command sends a pending CSR to the Nevis PKI’s certificate management system (ZMS). For this command to work, the following preconditions need to be matched:
ZmsBaseUrl in keybox.conf needs to be set, pointing to a working Nevis PKI ZMS system
The machine needs to be known in the ZMS system. By default, the fully qualified hostname, returned by the following command, is used:
getent hosts 'uname -n'
This value may be customized by setting the ServerName attribute in keybox.conf.
The common use of this command is:
$> neviskeybox uploadCDR -slot public -label node
Operation - Using nevisKeybox without an Administration Wrapper
The keybox command should not be used without the wrapper and always with the default path conventions to avoid any side-effects.
To use keybox directly from the shell, the keybox environment must be correctly configured to get the same behavior.
These preconditions are:
- keybox should be used as user root (it works with any other user too, but key material is supposed to be handled as restrictive as possible)
- OpenSSL needs to be in the PATH
- Java 1.8 needs to be in the PATH (java, keytool)
- The nevisKeybox keytool extension nevissectools.jar needs to be in the CLASSPATH
Creating a keybox Store
Create a keybox environment as follows:
$> mkdir -p /var/spool/keybox
$> cd /var/spool/keybox
$> chmod 755 .
$> cp /opt/neviskeybox/bin/nevissectools.jar ./
$> cp /opt/neviskeybox/bin/openssl ./
$> cp /opt/neviskeybox/bin/keybox ./
The nevisKeybox package contains nevissectools.jar and openssl as well as the keybox script.
Executing keybox Commands
Execute keybox commands as follows:
$> cd /var/spool/keybox
$> export CASSPATH=./nevissectools.jar
$> export PATH=$PATH:./:/usr/lib/java/jre/bin
$> ./keybox genreq
$> ./keybox import -file /tmp/some_cert.pem
Configuration
This chapter covers the configuration of nevisKeybox. It gives a detailed description of all technical elements.
The behavior of the nevisKeybox can be customized using neviskeybox config to edit the configuration file of the default instance (using vi).
Keybox, Key and CSR Attributes
The following attributes may be set:
Global keybox attributes
Attribute name | Type, Usage Constraints | Description |
Formats
- Type, Usage Constraints: set of supported formats, default: pem der pkcs8 pkcs12 jks
- Description: Specify the supported formats. In addition to the defaults, cert8 (for the Netscape certificate database format) is supported. JKS requires JRE 1.8.
DefaultSlot
- Type, Usage Constraints: identifier, default: default
- Description: Slot to use when -slot option is omitted.
DefaultLabel
- Type, Usage Constraints: identifier, default: node
- Description: Label to use when -label option is omitted.
DefaultTrustAccess
- Type, Usage Constraints: file access permission, default: 0644
- Description: Permissions applied to files, not containing any private key material. Files with key material always have permission 0600 (only read-/writeable by the file owner). By default with 0644 the file is read-/writeable by the file owner and readable by everyone. These are Linux file permissions in octal format, see man chmod for details.
DefaultSlotAccess
- Type, Usage Constraints: file access permission, default: 0755
- Description: Permission applied to new file system slots. Ensure to have execution bit is set, as this is required in Linux to access directories. By default the slot is read-/writeable by the file owner and readable by everyone. These are Linux file permissions in octal format, see man chmod for details.
ExpiryPeriod
- Type, Usage Constraints: integer
[days]
, default: 30 - Description: This period is used to check certificates. On execution of the check or renewExpiring command, this option is passed to the expiration checker. Certificates expiring within that period are shown.
- Type, Usage Constraints: integer
Key and CSR creation attributes
DefaultDays
- Type, Usage Constraints: integer
[days]
, default: 720 - Description: This attribute specifies the requested validity for certificate requests.
- Type, Usage Constraints: integer
DefaultKeySize
- Type, Usage Constraints: integer
[bits]
, default: 2048 - Description: This is the default key size used for all commands that generate key material without explicitly defining the key size.
- Type, Usage Constraints: integer
DefaultCountry
- Type, Usage Constraints: identifier, default: ch
- Description: The country name to use in the C attribute of CSRs.
DefaultState
- Type, Usage Constraints: identifier, default: undefined
- Description: The state name to use in the ST attribute of CSRs.
DefaultLocality
- Type, Usage Constraints: identifier, default: undefined
- Description: The locality name to use in the L attribute of CSRs.
DefaultOrg
- Type, Usage Constraints: identifier, default: company
- Description: The company name to use in the O attribute of CSRs. When finding the fully qualified host name of the machine, the command is executed on, the company is extracted as follows: host.company.tld => company
DefaultUnit0
- Type, Usage Constraints: identifier, default: undefined
- Description: The unit name to use in the first OU attribute of CSRs.
DefaultUnit1
- Type, Usage Constraints: identifier, default: undefined
- Description: The subunit name to use in the second OU attribute of CSRs.
DefaultCn
- Type, Usage Constraints: identifier, default: Fully qualified hostname
- Description: The company name to use in the CN attribute of CSRs. The automatically detected fully qualified hostname may have an unwanted value on multi-homed hosts or hosts without DNS. In this case, the machine’s identity may be set with this attribute.
DefaultCaDays
- Type, Usage Constraints: integer
[days]
, default: 10000 - Description: Validity period of the CA certificate generated with the createca command.
- Type, Usage Constraints: integer
DefaultCrlDays
- Type, Usage Constraints: integer
[days]
, default: 30 - Description: Validity period of the CRL generated with the gencrl command.
- Type, Usage Constraints: integer
Environment variables
PATH
- Description: Add e.g. /opt/adnjdk18/bin to point to a different JRE. |
CLASSPATH
- Description: Set e.g. /opt/keybox/lib/nevissectools.jar to add keybox utilities |
HSM Attributes
An HSM slot is mapped into nevisKeybox by specifying the following:
Attribute name | Type, Usage Constraints | Description |
---|---|---|
$slot | HSM slot section, replace $slot with the HSM slot name that you use in nevisKeybox. | |
pkcs11_slot | identifier, default: undefined | Physical HSM slot name, i.e. the name of the slot on the HSM. |
pkcs11_library | path, default: undefined | Path to the cryptoki library, offering the PKCS#11 driver. If this library depends on other shared libraries, the LD_LIBRARY_PATH may need to be enhanced. |
pkcs11_type | integer, default: undefined | Specific for Chrysalis / LunaSA (requires a value of 3). |
Chrysalis / LunaSA Sample Configuration
The following is a sample configuration for Chrysalis / Gemalto LunaSA for a nevisKeybox slot named hsm_partition1. On the file-system this slot would be located at /var/opt/neviskeybox/default/hsm_partition1/.
[hsm_partition1]
pkcs11_slot = partition1
pkcs11_library = /usr/lunasa/lib/libCryptoki2_64.so
# chrysalis specific
pkcs11_type = 3
Eracom Sample Configuration
A sample configuration for Eracom (LAN Box):
[eracom_openssl]
pkcs11_slot = openssl
pkcs11_library = /usr/eracom/libcryptoki.so
env:CT_SERVER = eracom
Use cases and Best practices
This chapter describes usage scenarios and configuration tasks related to PKI administration. A scenario consists of several administration or configuration tasks and may be part of a scenario that involves several Nevis components.
Administrators and integrators should be familiar with UNIX command line interfaces, the vi editor, and have a sufficient understanding of the Nevis architecture.
How to's
How to Create a Secure, Trusted Connection Between Two Nodes
This section provides details about how to set up a connection between two nodes (see connection between Reverse Proxy and Content Provider in Figure "Nevis sample architecture overview").
The set-up steps are:
if not done already, initialize neviskeybox for the first time
# on every node
$> neviskeybox handoverCreate a certificate request for node 1 (on node 1)
# on node1
$> neviskeybox certreq -slot default -label nodeCreate a certificate request for node 2 (on node 2)
# on node2
$> neviskeybox certreq -slot default -label nodeSubmit both certificate requests to the CA (using the CA’s web GUI or a mail client). You may manually post it by extracting the content of the certificate request and send it by mail or post it in a Web GUI:
# on node1 and node2:
$> cat /var/opt/neviskeybox/default/default/node_request.pemRetrieve the signed certificate and CA certificates from the CA (using the CA’s web GUI, or getting back an e-mail, etc.) and transfer them to both nodes.
Import the returned certificate(s) into keystore and CA certificates into truststore:
# on node1 and node2:
$> neviskeybox import -file /tmp/cert_bundle_from_ca.pem
Now configure the components according to 7.5, Nevis Configuration Reference, page 35.
Refer to "neviskeybox certreq" for command line option details. If you are using nevisKeybox in development or do not have an internal CA, refer to "neviskeybox sign".
How to Replace an Expiring Certificate
To replace a certificate that will soon expire, a new certificate needs to be generated (from the same CA that provided the existing one) to replace the existing one:
Get a new certificate by generating a new certificate request with the same keybox name (slot/label) as the existing one:
$> neviskeybox certreq -rekey -slot default -label node
Upload to generated CSR located at and let it sign by the CA.
Copy the resulting certificate back to the host.
Import the new certificate (will replace the old one):
$> neviskeybox import -file /tmp/new_cert.pem
To check for certificates which expire soon, see: neviskeybox check.
How to Use nevisKeybox with nevisProxy
More information about TLS configuration, servlets and SectokenVerifierCert can also be found in "nevisProxy Reference Guide: TLS Configuration", "nevisProxy Reference Guide: HttpsConnectorServlet" and "nevisProxy Reference Guide: SecToken verification".
Frontend TLS Connection
The frontend TLS connection is used by clients connecting to the nevisProxy, including browsers, mobile devices or other user agents. The TLS connection to nevisProxy is either 1-way SSL (server authentication) or 2-way SSL (client authentication).
It's recommended to always enable frontend TLS in production, especially for public facing services. Especially since some user agents (like Google Chrome or Firefox) start treating HTTP as insecure in 2018, leading to a bad end-user experience.
The server for the frontend TLS connection can be opened by executing: nevisproxy config server
Configuration Location and Attribute Name | nevisKeybox Suggestion | Remark |
---|---|---|
SSLCertificateFile | /var/opt/neviskeybox/default/public/node_keystore.pem | Required for 1-way SSL (server authentication). |
SSLCACertificateFile | /var/opt/neviskeybox/default/public/peer_truststore.pem | Required for 2-way SSL (client authentication). Client certificates are usually not provided by the same CA, therefore another truststore (containing all the trusted user CAs) is required. Add these using $> neviskeybox import -slot public -label peer . |
SSLCertificateChainFile | /var/opt/neviskeybox/default/public/node_keystore.pem | Optional: will be sent to client along with server certificate for verification. |
SSLPassPhrase | builtin | This is the default, it uses the passphrase passing mechanism as described in section Nevis Component Password Handling. The value builtin usually works out-of-the-box with certificate material stored in nevisKeybox. |
exec:/opt/nevisproxy/bin/keystorepwget | This is required when the keybox’s password generation mechanism is used (see "keybox certreq"). If the bc.security.PassPhraseDialog is set in bc.properties, this attribute may be left as builtin. |
SecToken Verification
The truststore, required to verify identity tokens from the authentication server is configured in the nevisProxy’s web.xml as follows (assuming two authentication server instances with different signer certificates):
<context-param>
<param-name>SectokenVerifierCert</param-name>
<param-value>
/var/opt/neviskeybox/default/nevis/truststore/certificate1.pem
/var/opt/neviskeybox/default/nevis/truststore/certificate2.pem
</param-value>
</context-param>
To directly open the web.xml in an editor, you can run: nevisproxy config
HttpsServletConnector
The HTTPS connector of nevisProxy is configured in the nevisProxy’s web.xml (using <init-param>
) as follows:
Configuration Location and Attribute Name | nevisKeybox Suggestion | Remark |
---|---|---|
SSLClientCertificateFile | /var/opt/neviskeybox/default/default/node_keystore.pem | Required for 2-way SSL (client authentication) |
SSLCACertificateFile | /var/opt/neviskeybox/default/default/node_truststore.pem | Required for 1-way SSL (server authentication) |
The passphrase getter is configured in the bc.properties file as follows: PassPhraseDialog=pipe:///opt/nevisproxy/bin/keystorepwget |
How to Use nevisKeybox with nevisAuth
For token signing, nevisAuth needs a signer certificate configured in the esauth4.xml configuration file (KeyObject element):
Configuration Location and Attribute Name | nevisKeybox Suggestion | Remark |
---|---|---|
certificate | /var/opt/neviskeybox/default/nevis/authSigner_keystore.jks?alias=authSigner | |
privateKey | /var/opt/neviskeybox/default/nevis/authSigner_keystore.jks?alias=authSigner | |
passPhrase | pipe:///opt/keybox/bin/keystorepwget /var/opt/neviskeybox/default/nevis/authSigner_keystore.jks?alias=authSigner , or (if no other password source should be checked): pipe:///var/opt/neviskeybox/default/nevis/authSigner_keypass | This is required when the keybox’s password generation mechanism is used (see "keybox certreq"). |
To open the esauth4.xml directly in an editor you can also run: nevisauth config.
The JSSE layer (for HTTPS connections to content providers, e.g., used by the distributed session management’s notification mechanism) is configured according to section "JSSE Client Set-up".
How to Use nevisKeybox with Java / JSSE
Java’s JSSE component, required for SSL network connections, needs to be configured by providing:
- A mandatory truststore (required to verify the server’s node certificate).
- An optional keystore (required when the server requires client authentication).
JSSE can only be configured once per class loader and the location of the key stores (either in JKS or P12 format) needs to be provided on the command line (JVM initialization) as follows:
-Djavax.net.ssl.trustStore=/var/opt/neviskeybox/default/default/node_truststore.jks
-Djavax.net.ssl.trustStorePassword=changeit
-Djavax.net.ssl.keyStore=/var/opt/neviskeybox/default/default/node_keystore.p12
-Djavax.net.ssl.keyStoreType=pkcs12
-Djavax.net.ssl.keyStorePassword=changeit
The passphrase for the truststore is optional as Java keystores may be read without it. But JCE is not able to verify its integrity without it.
How to Enable Nevis ZMS
Integration and support for the Nevis certificate management system (ZMS) is being removed in future versions of nevisKeybox.
To integrate nevisKeybox with the Nevis PKI (ZMS subsystem) for centralized certificate management, proceed as follows:
Check connectivity to ZMS:
$> neviskeybox downlodCDP
When you have problems, check for the following causes:
ZMS is not configured, see ZMS Integration Attributes]"
ZMS is not reachable over the network.
The machine is unknown in the ZMS system: Either the machine must be entered into the ZMS database or the ServerName attribute in the configuration must be adapted (required, when the hosts default host name is not equal to the public ZMS name).
The verification of the CDP fails because the ZMS truststore (usually slot zms, label cdpSigner, see keybox.conf for the configured truststore) does not match the ZMS signer key. You may import the correct truststore by executing:
$> neviskeybox import -slot zms -label cdpSigner \
-file /tmp/zms_truststore.pem
If there is already content in the keybox, proceed as described in neviskeybox extractCDP".
If the keybox is empty, CDP signatures are not verified upon download and installation to be able to bootstrap the keybox from ZMS. In this case, a warning is displayed and the administrator should verify, that the keybox content (especially the certificate, used to verify the CDP signature, usually installed in slot zms with label cdpSigner) is safe. If the CDP signer changes in the ZMS installation, the locally installed verification certificate needs to be removed to bootstrap the keybox trust again.
Password Handling
This chapter discusses the different approaches to handling passwords and passphrases for key material.
nevisKeybox Password Handling
Keystores usually ask for a passphrase when executing administration commands to manipulate or access private keys.
The following policies apply or are recommended:
- Passphrases may be set per new key object (slot/label), but if not required otherwise, it is recommended to apply a common HSM policy, in which case, the passphrase is set per slot (all objects in that slot having the same passphrase).
- When using user based access policies, file permissions on key objects are set to world-readable. The key is only protected by the passphrase in this case. If passphrase caching is used, the generated passphrase getter remains executable only for the keybox owner (root) and a user specific passphrase getter is generated and managed.
- When importing a protected private key into the keybox, the keybox object is protected by the same passphrase and may be changed with the "neviskeybox passwd" command.
- Keybox supports batch processing (-batch option), in which case, the passphrase needs to be set in the environment (PASSPHRASE variable). Do not forget to unset it or close the shell after keybox administration.
Nevis Component Password Handling
Private key material, which is stored in a file system or accesses a hardware security module (HSM), usually requires a password (either to decrypt a protected private key or to log in to the HSM). These passwords need to be provided statically (via the configuration), dynamically (through execution of a program), or interactively (entered by the user who starts a component).
As the interaction with a user is the only really secure way of bootstrapping a security system (as no information is persistently stored on the local machine), all Nevis components support password pre-fetching to allow interactively entering passwords, even for servers that detach from the controlling terminal.
This means, that each server on start-up requires a password for each protected PKI object configured. To provide this, the following mechanisms are possible:
Interactive Prefetch (default):
All key objects are scanned from the configuration and checked for passphrase requirements (this is done by the start-up wrapper of the Nevis component). If a passphrase is required, it is requested from the user’s terminal and checked against the key object. If the key object is accessible, an environment variable is set. The component starts and retrieves the passphrase. On first access, the environment variable is cleared from the environment.
Environment Preset:
Nevis components usually have an ‘op component config env’ command to set environment variables. For each key object, a corresponding environment variable may be set to disable the interactive passphrase getting (see: Setting Passphrase Environment Variables). Note that the permission of the environment file may be set to read/write-only for the installation owner.
Passphrase Getter:
Most Nevis components allow to configure a passphrase getter, which should act as follows: The passphrase getter should be executable for the installation owner (and the installation group in case the process runs under root and spawns childs, which change the process user, done e.g. by nevisProxy).
- The first command line argument refers to the certificate the passphrase is used for.
- The command should return the plain text passphrase on stdout and exit with status code.
Refer to sections "Using keystorepwget" for more details about passphrase pre-fetchers.
Credential Server (i.e. nevisCred):
Passphrases may be managed by a locally installed, secured credential server, which pre-fetches all passphrases from the administrator or a specially secured context at host start-up. This has the advantage that components may request passphrases without user interaction (requires OS authentication and an authorization mechanism in the credential server). In this case, Nevis components may configure the corresponding passphrase getter.
Only the default instance of nevisCred is supported, it's not possible to use another instance.
Application-Server Passphrase Handling
The passphrase caching mechanism is not used for application-servers, as the administration facilities of those servers usually provider their own mechanism for storing passphrases. The problem here remains. Passphrases are usually obfuscated and written to configuration-files or databases, but the obfuscation mechanism does not prevent hackers from finding out the correct value. The following rules apply:
- Remove obfuscated passphrase entries when sending around configuration files
- Carefully check file access to configuration files, containing obfuscated passphrases
- Consider using a passphrase vault as a separated service
Setting Passphrase Environment Variables
Password pre-fetching may be disabled by presetting the required environment variable. It is recommended to use passphrase getters (such as the ones nevisKeybox provides when using the "neviskeybox passwd -keep" command).
The rules for these variables are as follows:
HSM / Hardware tokens (PKCS#11):
If the string contains a pinenv attribute, set the environment variable specified there. If no pinenv is present, set the environment variable PKCS11_PIN.
You also need to set PKCS11_PIN_ENVIRONMENT_AHEAD=1.
Soft tokens (file-system):
The environment variable is a SHA1 hash of the certificate path, pre-fetched by ‘pp_’ and may be generated as follows:
$> echo /var/opt/neviskeybox/default/default/node_keystore.pem | \
/opt/neviskeybox/bin/openssl sha1 | \
awk '{ print "pp_" $1 }'
Using keystorepwget
This passphrase getter implements different search strategies to get passphrases.
The following process is applied:
- If the Nevis credential server (nevisCred) is installed, the server is contacted to return a passphrase. Only objects in the default instance of the nevisCred instance are supported.
- (for soft certificates only) If a key object named x.** or x_key.** has a correspondingx_keypassfile at the same location, this file is considered to be a passphrase getter, which (when executed) will return the passphrase. When using user based access policies and the passphrase getter is executed by useruser,x_user_keypassis tried, before usingx_keypass.*
- The environment is scanned for a passphrase according to the rules explained in Setting Passphrase Environment Variables.
If these do not meet the security requirements, a customized passphrase getter (program, script) needs to be provided meeting the specification in section "Nevis Component Password Handling".
HSM Support
The term HSM refers to Hardware Security Module (boxes, designed to store sensitive key material in a specially secured environment). The administration API for these boxes is usually very vendor specific.
The nevisKeybox command line API is designed to address HSM administration too (therefore the slot and label design), but does not currently provide specific integrations.