Appendix K - OOCD changes introduced with 7.2311.0.x

Together with Java 17 upgrade we rationalized and simplified the OOCD implementations impacting the API and configuration to be more in line with the Remote Session store.

Implementation

We removed the deprecated File system based OOCD in favour of the SQL-based implementation. The file based implementation is difficult to use and not fit for modern environments like Kubernetes. Additionally, even in "classic deployments" performance was limited by the operating system and trying to scale the peformance with shared file system setups often lead to performance downgrades.

• For production setups we recommend to use the SQL based implementation.
• For development or test environments the new in-memory implementation can be used to reduce integration time. This in-memory implementation must not be used for production setups.

Default behaviour

The OOCD is now optional in the configuration and there is no default implementation. (The file system based implementation was default before) In case no OOCD is configured and an attempt is made to use the OOCD, nevisAuth will throw an error.

caution

Note that this is a breaking change, because before the File based OOCD was used by default which required no special configuration.

The new implementation does not have a default implementation to raise awareness if OOCD is required for your setup or not. It could be argued that the in-memory implementation should be default, but that could cause customers accidentally / unaware using the in-memory implementation in production.

Configuration

The configuration of the OOCD was moved from the JAVA_OPTS in the env.conf to the esauth4.xml. As the OOCD is a crucial component of nevisAuth, relocating the configuration to the esauth4.xml increases consistency. The new configuration option is expected to be between the </SessionCoordinator> and the <AuthEngine> tags in the esauth4.xml file.

SQL based OOCD configuration

Old OOCD configuration example supplied via env.conf

-Dch.nevis.esauth.OutOfContextDataService.class=ch.nevis.esauth.ooc.sql.SqlOOCDService \
-Dch.nevis.esauth.OutOfContextDataService.jdbcUrl=jdbc:mariadb://localhost:3306/OOCD?autocommit=true \
-Dch.nevis.esauth.OutOfContextDataService.dataUser=oocddatauser \
-Dch.nevis.esauth.OutOfContextDataService.dataUserPassword=password \
-Dch.nevis.esauth.OutOfContextDataService.schemaUser=oocdschemauser \
-Dch.nevis.esauth.OutOfContextDataService.schemaUserPassword=password \
-Dch.nevis.esauth.OutOfContextDataService.automaticDbSchemaSetup=true"

New esauth4.xml based configuration

    ...
    </SessionCoordinator>

    <RemoteOutOfContextDataStore
        connectionUrl="jdbc:mariadb://localhost:3306/OOCD?autocommit=true"
        connectionUser="oocddatauser"
        connectionPassword="password"
        connectionSchemaUser="oocdschemauser"
        connectionSchemaPassword="password"
        connectionTimeout="30000"
        connectionMaxLifeTime="60000"
        connectionMaxPoolSize="20"
        connectionAutomaticDbSchemaSetup="true"
        reaperPeriod="60"/>

    <AuthEngine ...

Note that the examples above use default values, there is no need to specify all options. This is just done to demonstrate the differences.

New configuration options to match the options in the remote session store:

• connectionTimeout
• connectionMaxLifeTime
• connectionMaxPoolSize
• reaperPeriod

Change in password resolution

Since the configuration has moved into the esauth4.xml, you can use the Passwords in the configuration methods to resolve passwords.

The syntax ${exec:...} of the env.conf is not supported in the esauth4.xml file.

In-memory OOCD configuration

    ...
    </SessionCoordinator>

    <LocalOutOfContextDataStore
        reaperPeriod="60"/>

    <AuthEngine ...

API

The following table list the replacements for the removed deprecated methods:

Original method	Replacement method
String get(String path, Properties metadata)	Map<String, String> getWithPrefix(String keyPrefix) or String get(String key)
void set(String path, String value, Properties props, Date notOnOrAfter)	set(String key, String value, Instant notOnOrAfter) or void set(Map<String, String> keyValuePairs, Instant notOnOrAfter)
String getOrSet(String path, String value, Date notOnOrAfter)	get() + if it was empty set()
String getOrSet(String path, String value, Properties props, Date notOnOrAfter)	get() + if it was empty set()
List<String> list(String path)	Map<String, String> getWithPrefix(String keyPrefix)
Properties getMeta(String path)	Map<String, String> getWithPrefix(String keyPrefix)
String getMeta(String path, String key)	String get(String key)
void setMeta(String path, String key, String value)	set(String key, String value, Instant notOnOrAfter)
String removeMeta(String path, String key)	void remove(String key)
❌	void removeWithPrefix(String keyPrefix)

Multiple metadata special cases

In case you used the Meta named functions or provided any Properties metadata with multiple entries you likely need to do code change in your custom java or groovy scriptstate. For that see the section Multiple metadata below. Note that the SQL-based implementation already did not support these methods at all, so in case you used the SQL-based implementation your setup will likely not need adaptions in this case.

All OOCD interface methods after changes:

Set key-value pair(s):

• void set(String key, String value, Instant notOnOrAfter)
• void set(Map<String, String> keyValuePairs, Instant notOnOrAfter)

Get key-value pair(s):

• String get(String key)
• Map<String, String> getWithPrefix(String keyPrefix)

Remove key-value pair(s):

• void remove(String key)
• void removeWithPrefix(String keyPrefix)

ScriptState specific

The deprecated binding dataPersistenceService was removed. Originally it was introduced to replace the oocd binding, but over time we realized the oocd makes more sense as everybody is familiar with that. Therefore instead of the removed dataPersistenceService the oocd binding should be used.

File based OOCD data migration

By default, nevisAuth uses OOCD for the SAML and OAuth2 / OIDC flows. In case you use any of those you might consider migration existing data for smoother customer experience. In case you only use the OOCD in custom auth states or scripts, you have to decide based on your use case if it makes sense to migrate the data or not.

Migration script

Nevis provides the following python script to generate a MariaDB or PostgreSQL specific SQL insert script from existing file based OOCD data. Once the script is run, the SQL insert script have to be run manually on the target database for the replacement SQL implementation backend. There is no migration option for the in-memory implementation.

The script handles 2 use-cases:

• Singular key-value entry with no special metadata.
• Multiple metadata entries which will be written as individual key-value pairs. This could be relevant in case you used the OOCD from custom auth states or scripts. For more about this case, see the description below the script. (SAML falls into this case, which is handled automatically)

OOCD File to SQL migration

import argparse
import os
import fnmatch
import sys
import xml.etree.ElementTree as ET
from datetime import datetime

parser = argparse.ArgumentParser(description="Offline migration script for File based OOCD to SQL")
parser.add_argument('--path', type=str, required=True, help='Where key-values are stored. Previous dir configuration option of the FileSystemOOCDService')
parser.add_argument('--out', type=str, required=True, help='File where the SQL script is generated.')
parser.add_argument('--db_type', type=str, required=True, help='Syntax to use: mariadb or postgresql.')
args = parser.parse_args()

print('Running at:' + args.path)

if os.path.isfile(args.out):
    print('Output file already exists, please specify a different filename in --out.')
    sys.exit(1)

def mariadb():
    return args.db_type.lower() == 'mariadb'.lower()

def postgresql():
    return args.db_type.lower() == 'postgresql'.lower()


if not (mariadb() or postgresql()):
    print('Invalid db_type is used, supported is mariadb and postgresql.')
    sys.exit(1)

def find_files(directory, pattern):
    for root, dirs, files in os.walk(directory):
        for basename in files:
            if fnmatch.fnmatch(basename, pattern):
                filename = os.path.join(root, basename)
                yield filename

def get_key_from_file_path(path, root_path, extension):
    middle_path = path[len(root_path):]
    return middle_path.replace(extension, '')

def insert(key, value, expiration):
    if mariadb():
        value = value.replace("\\", "\\\\").replace("'", "''").replace("\0", "\\0").replace("\n", "\\n").replace("\r", "\\r").replace("@", "\\@")
        return f"INSERT INTO `nevisauth_out_of_context_data_service` (`key`, `value`, `reap_timestamp`) VALUES ('{key}', '{value}', '{expiration}') ON DUPLICATE KEY UPDATE `value`=VALUES(`value`), reap_timestamp=VALUES(reap_timestamp);"
    elif postgresql():
        value = value.replace("\\", "\\\\").replace("'", "''").replace("\0", "\\0").replace("\n", "\\n").replace("\r", "\\r")
        return f"INSERT INTO nevisauth_out_of_context_data_service (key, value, reap_timestamp) VALUES ('{key}', E'{value}', '{expiration}') ON CONFLICT (key) DO UPDATE SET value=EXCLUDED.value, reap_timestamp=EXCLUDED.reap_timestamp;"

with open(args.out, 'a') as output_file:
    for filename in find_files(args.path, '*.meta'):

        key = get_key_from_file_path(filename, args.path, '.meta')

        # load meta file
        tree = ET.parse(filename)
        root = tree.getroot()
        meta_props = {}

        for child in root:
            if 'key' in child.attrib:
                meta_key = child.attrib['key']

                if meta_key == '__NoOnOrAfter':
                    expiration = child.text
                    expiration = datetime.utcfromtimestamp(int(expiration)/1000)
                if not meta_key.startswith('__'):
                    meta_props[meta_key] = child.text

        # load data file
        path_without_extension, _ = os.path.splitext(filename)
        datafile = path_without_extension + '.data'
        if os.path.isfile(datafile):

            with open(datafile, 'r') as file:
                value = file.read()

            # we have data file with metadata properties
            if len(meta_props) > 0:
                with open(filename, 'r') as file:
                    meta_xml = file.read()

                print("Meta properties will amended as extra properties using the data key prefix: " + key)

                sqls = []
                sqls.append(insert(key, value, expiration))

                for meta_key, meta_value in meta_props.items():
                    sqls.append(insert(key + '/' + meta_key, meta_value, expiration))
                sqls = [s + '\n' for s in sqls]

                output_file.writelines(sqls)

            # single data file without additional metadata
            else:
                print("Single data without metadata: " + key)
                sql = insert(key, value, expiration)
                output_file.write(sql + '\n')
        else:
            print("Found no Data file. Incorrect entry. " + key)

Multiple metadata

You are most likely a user of multiple metadata entries if used the following deprecated functions in your custom auth state or scripts:

• String get(String path, Properties metadata)
• void set(String path, String value, Properties metadata, Date notOnOrAfter)
• String getOrSet(String path, String value, Properties metadata, Date notOnOrAfter)
• Properties getMeta(String path)

Or if you used the void setMeta(String path, String key, String value) method.

Having multiple metadata entries is considered a special case as the metadata properties are considered internal. The point of the metadata was not to store multiple key value pairs for the same key. It can occur when you use the methods mentioned above in the OOCD API. These methods can be used to store properties (basically a map with multiple key-value entries) in the metadata on the top of the key-value pair or directly. These methods could be misused to store multiple values as meta-data without any actual data. Having that in mind, the script will split the multi-metadata entries into one key-value pair per metadata entry, so with the new API you can read those one by one.

Example:

/path/key = myValue
/path/key/metaProperty1 = propertyValue1
/path/key/metaProperty2 = propertyValue2

In the internal SAML case an exception was made quite a long time ago to store all metadata as an xml in one key-value pair. This is now automatically split into multiple key-value pairs when such an entry is found by the production code.