Shared out-of-context data

In general, the state of authentication processing is limited to the session of a user. Any state changes done within the context of a request (or by extension, within the conversation), which are not stored in the user's session, are lost when the operation has completed. In any case, data stored in these contexts can only be seen by operations running within the same context, that is, by the same user.

However, in some cases it is required to maintain a global state, which is not scoped to a particular user session and which can have a (much) longer lifetime. For this purpose, nevisAuth uses its shared out-of-context data service (OOCDS).

For an overview of the respective use cases, see the section Use Cases further below.

The out-of-context data service is provided by the OutOfContextDataService interface. It has the following properties:

• Hierarchically structured data with unique entry keys similar to file system paths.
• Each entry has an associated expiration date and will not be visible after this date.
• Each entry can be associated with an arbitrary number of meta data in the form of key/value pairs.
• Entries can be queried and listed based on branches of the hierarchical structure, much like listing files in a file system directory.
• Reading, writing, removing and querying entries is guaranteed to remain consistent under load of multiple accessing threads and processes. However, no guarantees are made regarding the performance of those operations.

Use Cases

The OOCDS interface is used in several cases, among others:

• SAML:
• OAuth / OpenID Connect:
• Custom:

It can generally be assumed that federation protocols are using the OOCDS.

Configuration

To define the implementing class and to set up its instance, you can use system properties with the following syntax:

-Dch.nevis.esauth.OutOfContextDataService.class=fully.qualified.ClassName
-Dch.nevis.esauth.OutOfContextDataService.param1=aValue
-Dch.nevis.esauth.OutOfContextDataService.param2=anotherValue

By default, nevisAuth uses the FileSystemOOCDService for out-of-context data (see the section FileSystemOOCDService]. To disable this service, configure "off":

-Dch.nevis.esauth.OutOfContextDataService.class=off

FileSystemOOCDService

Fully qualified class name: ch.nevis.esauth.ooc.FileSystemOOCDService

This OOCD service class implements the OOCD interface using the file system as persistence medium and as locking mediator. Each data entry is stored in a simple file, its meta data is placed in a property file. The hierarchical structure of the data is translated into a directory structure.

Coordination of multiple processes is done using a single global file lock. Coordination of multiple threads within a process holding this lock is implemented with Java concurrent locks.

A reaper thread running in the background periodically deletes files whose data entries have expired.

Property	Type Usage Constraints	Description
dir	string (file system path)defaults to /var/opt/nevisauth/<instance>/run/data	This property sets the directory in which the data is persisted in the file system. To set up data sharing between multiple nevisAuth instances, configure the same path here.
reaperInterval	number (seconds)defaults to 60 seconds	This property sets the interval of the reaper thread that clears away outdated files. Note that outdated data will be seen by the service automatically, even if the files holding the data have not yet been removed by the reaper.
lockGloballyWhenReaping	boolean defaults to true	With this property, the locking behavior can be relaxed. By default or when set to "true", the filesystem directory is completely locked when entries are reaped. When set to "false", the file, which will be deleted, is locked locally, allowing better performance when the reaper runs.

SqlOOCDService

Fully qualified class name: ch.nevis.esauth.ooc.sql.SqlOOCDService

The SqlOOCDService dataservice uses an SQL database as a backend to store out-of-context data. Currently only MariaDB databases are supported.

It's highly recommended to migrate from both file and Couchbase data services to the SqlOOCDService dataservice!

Differences between SQL and file system based OOCDS

The SqlOOCDService out-of-context data service does not provide the following elements:

• Hierarchically structured data with unique entry keys similar to file system paths.
• Querying and listing entries based on branches of the hierarchical structure, like listing files in a file system directory.

Example:

• /myhome is the parent of /myhome/attr1 and /myhome/attr2 in CouchbaseOOCDService and FileSystemOOCDServiceimplementations.
• In SqlOOCDService, however, they are not related. You can set/remove values for the following keys independently: myhome, /myhome/attr1 and /
• Therefore listing/setting/removing /myhome will not list/set/remove /myhome/attr1 and /myhome/attr2.

Database setup

Possible approaches:

1. Create the schemaUser who has the rights to create the database table in a newly created database, then create the dataUser who can modify the content of this table.
2. You create the table by hand with the schemaUser.
3. You rely on nevisAuth to use the schemaUserto create the table.
4. Create the database table and the dataUser with appropriate rights using an existing administrator user.
5. Use an existing database.
6. Create a new database by hand.
7. Now create the database schema. If you want to store strings containing special characters, your database must use a charset supporting these special characters (e.g. UTF-8).

CREATE DATABASE IF NOT EXISTS OOCD CHARACTER SET ='utf8' COLLATE ='utf8_unicode_ci';

4. Now create users to connect to the database.

5. The following example works for local connections:

   CREATE USER IF NOT EXISTS `oocdschemauser`@`localhost` IDENTIFIED BY 'password';
   GRANT CREATE ON OOCD.* TO `oocdschemauser`@`localhost`;
   
   CREATE USER IF NOT EXISTS `oocddatauser`@`localhost` IDENTIFIED BY 'password';
   GRANT SELECT, INSERT, UPDATE, DELETE ON OOCD.* TO `oocddatauser`@`localhost`;
   
   FLUSH PRIVILEGES;

6. For remote connections use something like this:

   CREATE USER IF NOT EXISTS `oocdschemauser`@`%` IDENTIFIED BY 'password';
   GRANT CREATE ON OOCD.* TO `oocdschemauser`@`%`;
   
   CREATE USER IF NOT EXISTS `oocddatauser`@`%` IDENTIFIED BY 'password';
   GRANT SELECT, INSERT, UPDATE, DELETE ON OOCD.* TO `oocddatauser`@`%`;
   
   FLUSH PRIVILEGES;

Depending on your preferences, you can also re-use the NSS database and nss_auth user from the Remote Session Store setup "). Note that in this case still create the user that will create the table, or you have to create the table manually by an administrator user.

It is not recommended using the same user for database table creation and data modification.

By default, the SqlOOCDService automatically creates the required database table in the SQL database (can be disabled by setting the automaticDbSchemaSetup to "false"). The code below shows the current default table definition:

SQL Out of Context Data Service Table

CREATE TABLE IF NOT EXISTS `nevisauth_out_of_context_data_service` (
 `key` VARCHAR(1024) NOT NULL,
 `value` MEDIUMTEXT NOT NULL,
 `reap_timestamp` TIMESTAMP NOT NULL,
 PRIMARY KEY ( `key`),
 INDEX reap_timestamp_idx (`reap_timestamp`)
);

You can adapt the table above according to your needs, if your data exceeds the limits defined above (key size of 1024 bytes and storage size of 16 MB/MEDIUMTEXT), or fits into smaller data storage types like TINYTEXT or TEXT. Note that use the same table and column names.

Note that the key column size is 1024, but special characters can reduce the storage size.

MariaDB has a maximum index size limitation at 3072 bytes. This can vary based on db page size settings, see `http://mariadb.com/kb/en/innodb-limitations/#page-sizes.

In an unlikely special case, where you only store 4 byte characters like emojis, you can only store 768 characters.

Configuring nevisAuth

The SqlOOCDService can be configured by JVM command line options. The following table depicts the available configuration options:

Property	TypeUsage Constraints	Description
jdbcUrl	string JDBC URL	The JDBC URL to the MariaDB database. For more details regarding the syntax, see the MariaDB documentation.
dataUser	string example value: data-user	The username required to access the data in the database. This user must have SELECT, INSERT, UPDATE and DELETE access rights to the database.
dataUserPassword	string	The password of the user accessing the data.
schemaUser	string	The name of the user that creates the schema and tables in the database. This user must have CREATE access rights to the database. If not provided, the system will use the user specified with the attribute user to create the schema.It is recommended that separate users create the schema and access the data. You specify these users in the DB properties schemaUser and dataUser, respectively.
schemaUserPassword	string	The password of the user who creates the schema and the tables in the database.If not provided, the system will use the password specified with attribute passwordto create the schema.
automaticDbSchemaSetup	boolean default value: true	If set to "true", nevisAuth will automatically try to create the table used to store the data (with the CREATE TABLE IF NOT EXISTS syntax, as shown in the sample code snippet above).Set this property to "false", if you want to handle this differently, for example because you have different data sizing requirements. Also set the property to "false", if you did not specify the schemaUseror if the specified user does not have the required CREATE access rights.

The next code block shows an example configuration to be added in the Java options "):

-Dch.nevis.esauth.OutOfContextDataService.class=ch.nevis.esauth.ooc.sql.SqlOOCDService
-Dch.nevis.esauth.OutOfContextDataService.jdbcUrl=jdbc:mariadb://localhost:3306/OOCD
-Dch.nevis.esauth.OutOfContextDataService.dataUser=oocddatauser
-Dch.nevis.esauth.OutOfContextDataService.dataUserPassword=password
-Dch.nevis.esauth.OutOfContextDataService.schemaUser=oocdschemauser
-Dch.nevis.esauth.OutOfContextDataService.schemaUserPassword=password
-Dch.nevis.esauth.OutOfContextDataService.automaticDbSchemaSetup=true

Below, find another example where we created the database table manually and reused the NSS database and the nss_auth user from the remote session store . Note that the schemaUser falls back to the dataUser. So if the dataUser has no CREATE rights, you have to disable the property automaticDbSchemaSetup (see the SqlOOCDService - Properties].

-Dch.nevis.esauth.OutOfContextDataService.class=ch.nevis.esauth.ooc.sql.SqlOOCDService
-Dch.nevis.esauth.OutOfContextDataService.jdbcUrl=jdbc:mariadb://localhost:3306/NSS
-Dch.nevis.esauth.OutOfContextDataService.dataUser=nss_auth
-Dch.nevis.esauth.OutOfContextDataService.dataUserPassword=password
-Dch.nevis.esauth.OutOfContextDataService.automaticDbSchemaSetup=false

You may use the following syntax to avoid hard-coding passwords in a plain text file (Standalone deployment only):

Syntax	Example	Remarks
${exec:command}	dataUserPassword: ${exec:/var/opt/keys/own/instance/keypass.sh}	Executes the given command and uses its output as the value.
${env:variablename}	dataUser: ${env:DATA_USER}	Uses the value of the specified environment variable.

Resilient database setup

This chapter describes how to set up the SqlOOCDService in order for it to be tolerant towards database or network outages between nevisAuth and MariaDB. At least one database node must be available to prevent application failure.

Use cases

FailureOn the primary DB node, failure connections will move to the secondary DB node when a 30 second timeout expires, or immediately if there is an incoming request to the DB. You may experience increased response time for the duration of the switch.

RecoveryOn the primary DB node, recovery connections will move back to the primary DB node once the connection maximum lifetime expires. This is after 30 minutes.

Implementation overview

Regular clustering solutions provide all fault-tolerant features themselves. Therefore, the connecting application has no clue about the resilient setup. The suggested solution takes a different approach. In this setup, the connecting application becomes part of the resilient setup in terms of configuration.

Two key features to achieve resilience

• Connectivity (MariaDB JDBC driver)

  jdbc:mariadb:sequential//host-db1:3306,host-db2:3306/OOCD

• Data consistency (MariaDB replication)

Replication is done by the database. The application and the JDBC driver are not aware of it at all.

Replication

Overview of database users

The replicated OOCD is managed by several database users to separate concerns. The creation of the users is explained below.

Username	Purpose	Required permissions
replication_user	Account used by the slave node to log into the master node.	REPLICATION SLAVE, to be able to replicate data.
binarylog_user	Account used for binary log management.	SUPER permission, to be able to purge the binary logs.

Step-by-step setup of the replicated session store

This chapters assumes that the DB setup is already completed.

1. Creation of the replication user:

CREATE USER IF NOT EXISTS replication_user IDENTIFIED BY 'replicationpassword';
GRANT REPLICATION SLAVE ON *.* TO replication_user;

2. Creation of the binary logs user:

CREATE USER IF NOT EXISTS binarylog_user IDENTIFIED BY 'binarylogspassword';
GRANT SUPER ON *.* TO binarylog_user;

3. Configuration of the MariaDB service. To configure the MariaDB service, add the following entries to the file /etc/my.cnf as super user. The two configuration files (host-db1 and host-db2) differ at some points. The different lines are marked with (*).

   • Configure the MariaDB service on host-db1:

Configuration of MariaDB service on host-db1

[mariadb]
# Enabling binary log
log-bin
# The ID of this master (*)
server_id=1
# The ID of the replication stream created by this master (*)
gtid-domain-id=1
# The basename and format of the binary log
log-basename=mariadbmaster
binlog-format=MIXED
# Setting which tables are replicated
replicate_wild_do_table="OOCD.nevisauth_out_of_context_data_service"
# Avoiding collisions of primary IDs for tables where the primary ID is auto-incremented
# Auto-increment value
auto_increment_increment=2
# Auto-increment offset (*)
auto_increment_offset=1
# Suppressing duplicated keys errors for multi-master setup
slave_exec_mode=IDEMPOTENT
# Ignoring some data definition language errors
slave-skip-errors=1452, 1062
# Suppressing binary logs after a delay regardless of the replication status
expire_logs_days=1
# Maximum number of connections
max_connections=1000
# Size of each of the binary log files (default: 1GB)
max_binlog_size=500M
# Enabling writing to the DB in parallel threads for the replication
slave-parallel-threads=10
# enabling semi-synchronous replication
rpl_semi_sync_master_enabled=ON
rpl_semi_sync_slave_enabled=ON
# change to READ COMMITTED
transaction-isolation=READ-COMMITTED

• Configure the MariaDB service on host-db2:

Configuration of MariaDB service on host-db2

[mariadb]
# Enabling binary log
log-bin
# The ID of this master (*)
server_id=2
# The ID of the replication stream created by this master (*)
gtid-domain-id=2
# The basename and format of the binary log
log-basename=mariadbmaster
binlog-format=MIXED
# Setting which tables are replicated
replicate_wild_do_table="OOCD.nevisauth_out_of_context_data_service"
# Avoiding collisions of primary IDs for tables where the primary ID is auto-incremented
# Auto-increment value
auto_increment_increment=2
# Auto-increment offset (*)
auto_increment_offset=2
# Suppressing duplicated keys errors for multi-master setup
slave_exec_mode=IDEMPOTENT
# Ignoring some data definition language errors
slave-skip-errors=1452, 1062
# Suppressing binary logs after a delay regardless of the replication status
expire_logs_days=1
# Maximum number of connections
max_connections=1000
# Size of each of the binary log files (default: 1GB)
max_binlog_size=500M
# Enabling writing to the DB in parallel threads for the replication
slave-parallel-threads=10
# enabling semi-synchronous replication
rpl_semi_sync_master_enabled=ON
rpl_semi_sync_slave_enabled=ON
# change to READ COMMITTED
transaction-isolation=READ-COMMITTED

• Restart the MariaDB servers on both hosts:

  sudo service mariadb restart

Semi-synchronous replication

By default, MariaDB uses asynchronous replication. To reach more consistency, it is recommended using semi-synchronous replication. The database configurations previously shown enable semi-synchronous replication with the following lines:

rpl_semi_sync_master_enabled=ON
rpl_semi_sync_slave_enabled=ON

MariaDB versions before 10.3.3 require the installation of plug-ins for semi-synchronous replication and are not supported.Start-up of the replication

To start the replication, log in as root into your MariaDB client and run the following commands:

• on host-db1 (master is host-db2):

Setting master on host-db1

CHANGE MASTER TO
 MASTER_HOST='host-db2',
 MASTER_USER='replication_user',
 MASTER_PASSWORD='replicationpassword',
 MASTER_PORT=3306,
 MASTER_USE_GTID=current_pos,
 MASTER_CONNECT_RETRY=10;

• on host-db2 (master is host-db1):

Setting master on host-db2

CHANGE MASTER TO
 MASTER_HOST='host-db1',
 MASTER_USER='replication_user',
 MASTER_PASSWORD='replicationpassword',
 MASTER_PORT=3306,
 MASTER_USE_GTID=current_pos,
 MASTER_CONNECT_RETRY=10;

• on host-db1:

Starting replication on host-db1

START SLAVE;

• on host-db2:

Starting replication on host-db2

START SLAVE;

Additional setup

Purging the binary logs

With the provided configuration (expire_logs_days=1 in the MariaDB settings), the system will automatically remove the binary logs that are older than one day, even if the logs were not copied by the slave. This prevents the disk of the master node from being filled up in case the slave is down for a long time. The automatic binary log removal takes place when

• the master DB node starts,
• the logs are flushed (nevisAuth does not use this feature),
• the binary log rotates, or
• the binary logs are purged manually (see below).

So binary logs older than one day may exist, if none of the listed actions occurred recently.

Complementary to this expiration feature, MariaDB provides the possibility to manually purge the binary logs. The purge action removes all binary logs that were already copied by the slave. This allows a safe removal of the binary logs on a regular basis. The nevisProxy package is delivered with an adaptable purging script, which is located at: /opt/nevisproxy/sql/mariadb/purgebinarylogs.sh

To use this script,

• copy the script to a location of your choice, and
• adapt it to your configuration.

The script takes care of both DB nodes, so that it only needs to be configured once.

Note that if different database server nodes are used for nevisProxy and nevisAuth, you have set them up separately.

You can schedule the script to run for example once per hour, with a cron job:

/etc/crontab

0 * * * * /var/opt/nevisproxy/instance/conf/purgebinarylogs.sh # Absolute path of your adapted script

Size the binary logs

The provided configuration (max_binlog_size=500M in the MariaDB settings) allows you to configure the maximum size of the binary log files before rotating. The smaller the size, the more often rotations will occur, which will slow down replication. The advantage is a more efficient purge process. The bigger the size, the less often rotations will occur, but the disk may be filled with old logs.

According to our experiences, a size less than 8K does stop replication completely under heavy load, because the slave keeps rotating the logs.

Troubleshooting

Usually the slave stops the replication if an error occurs. You can check the state of the slave with the following SQL command:

show slave status\G

Note that showing the slave status requires the REPLICATION CLIENT grant.

If the replication has stopped, usually the error that caused it will be displayed. First you should try to fix the error. If this is not possible, you can do a "forced" restart of the slave like this:

• On the master call (to display the current state of the master):

MariaDB> show master status\G
*************************** 1. row ***************************
 File: mariadbmaster-bin.000131
 Position: 194630804
 Binlog_Do_DB:
Binlog_Ignore_DB:
1 row in set (0.00 sec)

• On the slave (using the values returned by the call "show master status\G" on the master):

STOP SLAVE;
CHANGE MASTER TO
 MASTER_LOG_FILE='mariadbmaster-bin.000131',
 MASTER_LOG_POS=194630804;
START SLAVE;

This way, the system restart the slave, without replicating to the slave all the sessions that occurred since the replication stopped.