Shared out-of-context data

In general, the state of authentication processing is limited to the session of a user. Any state changes done within the context of a request (or by extension, within the conversation), which are not stored in the user's session, are lost when the operation has completed. In any case, data stored in these contexts can only be seen by operations running within the same context, that is, by the same user.

However, in some cases it is required to maintain a global state, which is not scoped to a particular user session and which can have a (much) longer lifetime. For this purpose, nevisAuth uses its shared out-of-context data service (OOCDS).

The out-of-context data service is provided by the OutOfContextDataService interface. It has the following properties:

• Hierarchically structured data with unique entry keys similar to file system paths.
• Each entry has an associated expiration date and will not be visible after this date.
• Each entry can be associated with an arbitrary number of metadata in the form of key/value pairs.
• Entries can be queried and listed based on branches of the hierarchical structure, much like listing files in a file system directory.
• Reading, writing, removing and querying entries is guaranteed to remain consistent under load of multiple accessing threads and processes. However, no guarantees are made regarding the performance of those operations.

Use Cases

The OOCDS interface is used in several cases, among others:

• SAML

  • To verify the uniqueness of received SAML messages.
  • To look up a SAML artifact when an ArtifactResolve message has been received.

• OAuth / OpenID Connect

  • During the Authorization Code Flow.

• Custom

  • The ScriptState exposes this service via the scope oocd.
  • nevisAuth expressions may access oocd.

info

It can generally be assumed that federation protocols are using the OOCDS.

Configuration

To define the implementing class and to set up its instance, you can use system properties with the following syntax:

-Dch.nevis.esauth.OutOfContextDataService.class=fully.qualified.ClassName
-Dch.nevis.esauth.OutOfContextDataService.param1=aValue
-Dch.nevis.esauth.OutOfContextDataService.param2=anotherValue

By default, nevisAuth uses the FileSystemOOCDService for out-of-context data. To disable this service, configure "off":

-Dch.nevis.esauth.OutOfContextDataService.class=off

FileSystemOOCDService

Fully qualified class name: ch.nevis.esauth.ooc.FileSystemOOCDService

This OOCD service class implements the OOCD interface using the file system as persistence medium and as locking mediator. Each data entry is stored in a simple file, its meta data is placed in a property file. The hierarchical structure of the data is translated into a directory structure.

Coordination of multiple processes is done using a single global file lock. Coordination of multiple threads within a process holding this lock is implemented with Java concurrent locks.

A reaper thread running in the background periodically deletes files whose data entries have expired.

The properties of the file system OOCD are as follows:

• dir (string)

  Default value: /var/opt/nevisauth/<instance>/run/data

  Sets the directory in which the data is persisted in the file system. To set up data sharing between multiple nevisAuth instances, configure the same path here.

• reaperInterval (integer, seconds)

  Default value: 60

  Sets the interval of the reaper thread that clears away outdated files. Note that outdated data will be seen by the service automatically, even if the files holding the data have not yet been removed by the reaper.

• lockGloballyWhenReaping (boolean)

  Default value: true

  With this property, the locking behavior can be relaxed. By default or when set to "true", the filesystem directory is completely locked when entries are reaped. When set to "false", the file, which will be deleted, is locked locally, allowing better performance when the reaper runs.

SqlOOCDService

Fully qualified class name: ch.nevis.esauth.ooc.sql.SqlOOCDService

The SqlOOCDService dataservice uses an SQL database as a backend to store out-of-context data. Currently, MariaDB and PostgreSQL databases are supported.

caution

Migrate from the file data services to the SqlOOCDService data service.

Differences between SQL and file system based OOCDS

The SqlOOCDService out-of-context data service does not provide the following elements:

• Hierarchically structured data with unique entry keys similar to file system paths.
• Querying and listing entries based on branches of the hierarchical structure, like listing files in a file system directory.

Example:

• /myhome is the parent of /myhome/attr1 and /myhome/attr2 in FileSystemOOCDServiceimplementations.
• In SqlOOCDService, however, they are not related. You can set or remove values for the following keys independently: myhome, /myhome/attr1 and /
• Therefore listing, setting or removing /myhome will not list, set or remove /myhome/attr1 and /myhome/attr2.

Database setup

Possible approaches:

1. Create the schemaUser who has the rights to create the database table in a newly created database, then create the dataUser who can modify the content of this table.
   1. You create the table by hand with the schemaUser.
   2. You rely on nevisAuth to use the schemaUserto create the table.
2. Create the database table and the dataUser with appropriate rights using an existing administrator user.
   1. Use an existing database.
   2. Create a new database by hand.
3. Now create the database schema. If you want to store strings containing special characters, your database must use a charset supporting these special characters (e.g. UTF-8).
4. Now create users to connect to the database.

Depending on your preferences, you can also re-use the NSS database and nss_auth user from the Remote Session Store setup (see chapter Session management. Note that in this case still create the user that will create the table, or you have to create the table manually by an administrator user.

caution

It is not recommended using the same user for database table creation and data modification.

By default, the SqlOOCDService automatically creates the required database table in the SQL database (can be disabled by setting the automaticDbSchemaSetup to "false"). The code below shows the current default table definition together with the initial setup:

MariaDB

SQL Out of Context Data Service Table

CREATE DATABASE IF NOT EXISTS OOCD CHARACTER SET ='utf8' COLLATE ='utf8_unicode_ci';

CREATE USER IF NOT EXISTS `oocdschemauser`@`localhost` IDENTIFIED BY 'password';
GRANT CREATE ON OOCD.* TO `oocdschemauser`@`localhost`;

CREATE USER IF NOT EXISTS `oocddatauser`@`localhost` IDENTIFIED BY 'password';
GRANT SELECT, INSERT, UPDATE, DELETE ON OOCD.* TO `oocddatauser`@`localhost`;

FLUSH PRIVILEGES;

CREATE TABLE IF NOT EXISTS `nevisauth_out_of_context_data_service` (
    `key`            VARCHAR(1024) NOT NULL,
    `value`          MEDIUMTEXT    NOT NULL,
    `reap_timestamp`  TIMESTAMP(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6),
    PRIMARY KEY ( `key`),
    INDEX reap_timestamp_idx (`reap_timestamp`)
);

info

To allow remote connection you can replace the localhost with % to allow any host or with a specific host.

You can adapt the table above according to your needs, if your data exceeds the limits defined above (key size of 1024 bytes and storage size of 16 MB/MEDIUMTEXT), or fits into smaller data storage types like TINYTEXT or TEXT. Note that use the same table and column names.

caution

Note that the key column size is 1024, but special characters can reduce the storage size.

MariaDB has a maximum index size limitation at 3072 bytes. This can vary based on db page size settings.

In an unlikely special case, where you only store 4 byte characters like emojis, you can only store 768 characters.

PostgreSQL

SQL Out of Context Data Service Table

CREATE USER "oocdschemauser" WITH encrypted password 'password';
CREATE USER "oocddatauser" WITH encrypted password 'password';

CREATE DATABASE oocd
 WITH
OWNER = "oocdschemauser";

ALTER ROLE "oocdschemauser" IN DATABASE oocd SET search_path TO "oocdschemauser";
ALTER ROLE "oocddatauser" IN DATABASE oocd SET search_path TO "oocdschemauser";

\connect oocd "oocdschemauser";
CREATE SCHEMA "oocdschemauser" AUTHORIZATION "oocdschemauser"

GRANT USAGE ON SCHEMA "oocdschemauser" to "oocddatauser";
GRANT CONNECT ON DATABASE oocd TO "oocddatauser";

ALTER DEFAULT PRIVILEGES FOR USER "oocdschemauser" IN SCHEMA "oocdschemauser" GRANT SELECT, INSERT, UPDATE, DELETE, TRIGGER ON TABLES TO "oocddatauser";
ALTER DEFAULT PRIVILEGES FOR USER "oocdschemauser" IN SCHEMA "oocdschemauser" GRANT USAGE, SELECT ON SEQUENCES TO "oocddatauser";
ALTER DEFAULT PRIVILEGES FOR USER "oocdschemauser" IN SCHEMA "oocdschemauser" GRANT EXECUTE ON FUNCTIONS TO "oocddatauser";

CREATE TABLE IF NOT EXISTS nevisauth_out_of_context_data_service (
    key VARCHAR(1024) NOT NULL PRIMARY KEY,
    value TEXT NOT NULL,
    reap_timestamp TIMESTAMP WITH TIME ZONE NOT NULL
);
CREATE INDEX IF NOT EXISTS oocd_reap_timestamp_idx ON nevisauth_out_of_context_data_service (reap_timestamp);

Configuring nevisAuth

The SqlOOCDService can be configured by JVM command line options.

info

We highly recommend to use the exec statement to prevent passwords showing up in the shell history when using the ps command, see the following example:

-Dch.nevis.esauth.OutOfContextDataService.dataUserPassword=${exec:/var/opt/nevisauth/default/conf/credentials/dbPassword}

The dbPassword needs to be an executable script which echoes the password.

The following list depicts the available configuration options:

• jdbcUrl (string)

  The JDBC URL to the MariaDB / PostgreSQL database. For more details regarding the syntax, see the MariaDB documentation and the PostgreSQL documentation.

  info

  nevisAuth relies on the autocommit feature of the database.

  • MariaDB requires to enable it on the database level or configure it in the JDBC driver url in this property using the query parameter autocommit=true.
  • PostgreSQL by default have autocommit enabled. :::

• dataUser (string)

  The username required to access the data in the database. This user must have SELECT, INSERT, UPDATE and DELETE access rights to the database.

• dataUserPassword (string)

  The password of the user accessing the data.

• schemaUser (string)

  The name of the user that creates the schema and tables in the database. This user must have CREATE access rights to the database. If not provided, the system will use the user specified with the attribute user to create the schema.

  info

  It is recommended that separate users create the schema and access the data. You specify these users in the DB properties schemaUser and dataUser, respectively.

• schemaUserPassword (string)

  The password of the user who creates the schema and the tables in the database. If not provided, the system will use the password specified with attribute password to create the schema.

• automaticDbSchemaSetup (boolean)

  Default value: true

  If set to "true", nevisAuth will automatically try to create the table used to store the data (with the CREATE TABLE IF NOT EXISTS syntax, as shown in the sample code snippet above).

  Set this property to "false", if you want to handle this differently, for example because you have different data sizing requirements. Also set the property to "false", if you did not specify the schemaUser or if the specified user does not have the required CREATE access rights.

The next code block shows an example configuration to be added in the Java options in the section Server command-line interface:

-Dch.nevis.esauth.OutOfContextDataService.class=ch.nevis.esauth.ooc.sql.SqlOOCDService
-Dch.nevis.esauth.OutOfContextDataService.jdbcUrl=jdbc:mariadb://localhost:3306/OOCD
-Dch.nevis.esauth.OutOfContextDataService.dataUser=oocddatauser
-Dch.nevis.esauth.OutOfContextDataService.dataUserPassword=password
-Dch.nevis.esauth.OutOfContextDataService.schemaUser=oocdschemauser
-Dch.nevis.esauth.OutOfContextDataService.schemaUserPassword=password
-Dch.nevis.esauth.OutOfContextDataService.automaticDbSchemaSetup=true

Below, find another example where we created the database table manually and reused the NSS database and the nss_auth user from the remote session store. Note that the schemaUser falls back to the dataUser. So if the dataUser has no CREATE rights, you have to disable the property automaticDbSchemaSetup.

-Dch.nevis.esauth.OutOfContextDataService.class=ch.nevis.esauth.ooc.sql.SqlOOCDService
-Dch.nevis.esauth.OutOfContextDataService.jdbcUrl=jdbc:postgresql://localhost:5432/nss
-Dch.nevis.esauth.OutOfContextDataService.dataUser=nss_auth
-Dch.nevis.esauth.OutOfContextDataService.dataUserPassword=password
-Dch.nevis.esauth.OutOfContextDataService.automaticDbSchemaSetup=false

You may use the following syntax to avoid hard-coding passwords in a plain text file:

• ${exec:command}

  Example: dataUserPassword: ${exec:/var/opt/keys/own/instance/keypass.sh}

  Executes the given command and uses its output as the value.

• ${env:variablename}

  Example: dataUser: ${env:DATA_USER}

  Uses the value of the specified environment variable.

Resilient database setup using MariaDB

This chapter describes how to set up the SqlOOCDService in order for it to be tolerant towards database or network outages between nevisAuth and MariaDB. At least one database node must be available to prevent application failure.

Use cases

Failure

On the primary DB node, failure connections will move to the secondary DB node when a 30 second timeout expires, or immediately if there is an incoming request to the DB. You may experience increased response time for the duration of the switch.

Recovery

On the primary DB node, recovery connections will move back to the primary DB node once the connection maximum lifetime expires. This is after 30 minutes.

Implementation overview

Regular clustering solutions provide all fault-tolerant features themselves. Therefore, the connecting application has no clue about the resilient setup. The suggested solution takes a different approach. In this setup, the connecting application becomes part of the resilient setup in terms of configuration.

Two key features to achieve resilience

• Connectivity (MariaDB JDBC driver)

  • Configure a JDBC URL, where you define the DB nodes in priority order:

    jdbc:mariadb:sequential//host-db1:3306,host-db2:3306/OOCD

• Data consistency (MariaDB replication)

  • Configure a master-to-master replication.

info

Replication is done by the database. The application and the JDBC driver are not aware of it at all.

Replication

Overview of database users

The replicated OOCD is managed by several database users to separate concerns. The creation of the users is explained below.

• replication_user

  Required permissions: REPLICATION SLAVE, to be able to replicate data.

  Account used by the slave node to log into the master node.

• binarylog_user

  Required permissions: SUPER permission, to be able to purge the binary logs.

  Account used for binary log management.

Step-by-step setup of the replicated session store

This chapters assumes that the DB setup is already completed.

1. Creation of the replication user:

   CREATE USER IF NOT EXISTS replication_user IDENTIFIED BY 'replicationpassword';
   GRANT REPLICATION SLAVE ON *.* TO replication_user;

2. Creation of the binary logs user:

   CREATE USER IF NOT EXISTS binarylog_user IDENTIFIED BY 'binarylogspassword';
   GRANT SUPER ON *.* TO binarylog_user;

3. Configuration of the MariaDB service.

   To configure the MariaDB service, add the following entries to the file /etc/my.cnf as super user. The two configuration files (host-db1 and host-db2) differ at some points. The different lines are marked with (*).

• Configure the MariaDB service on host-db1:

  [mariadb]
  # Enabling binary log
  log-bin
  # The ID of this master (*)
  server_id=1
  # The ID of the replication stream created by this master (*)
  gtid-domain-id=1
  # The basename and format of the binary log
  log-basename=mariadbmaster
  binlog-format=MIXED
  # Setting which tables are replicated
  replicate_wild_do_table="OOCD.nevisauth_out_of_context_data_service"
  # Avoiding collisions of primary IDs for tables where the primary ID is auto-incremented
  # Auto-increment value
  auto_increment_increment=2
  # Auto-increment offset (*)
  auto_increment_offset=1
  # Suppressing duplicated keys errors for multi-master setup
  slave_exec_mode=IDEMPOTENT
  # Ignoring some data definition language errors
  slave-skip-errors=1452, 1062
  # Suppressing binary logs after a delay regardless of the replication status
  expire_logs_days=1
  # Maximum number of connections
  max_connections=1000
  # Size of each of the binary log files (default: 1GB)
  max_binlog_size=500M
  # Enabling writing to the DB in parallel threads for the replication
  slave-parallel-threads=10
  # enabling semi-synchronous replication
  rpl_semi_sync_master_enabled=ON
  rpl_semi_sync_slave_enabled=ON
  # change to READ COMMITTED
  transaction-isolation=READ-COMMITTED

• Configure the MariaDB service on host-db2:

  [mariadb]
  # Enabling binary log
  log-bin
  # The ID of this master (*)
  server_id=2
  # The ID of the replication stream created by this master (*)
  gtid-domain-id=2
  # The basename and format of the binary log
  log-basename=mariadbmaster
  binlog-format=MIXED
  # Setting which tables are replicated
  replicate_wild_do_table="OOCD.nevisauth_out_of_context_data_service"
  # Avoiding collisions of primary IDs for tables where the primary ID is auto-incremented
  # Auto-increment value
  auto_increment_increment=2
  # Auto-increment offset (*)
  auto_increment_offset=2
  # Suppressing duplicated keys errors for multi-master setup
  slave_exec_mode=IDEMPOTENT
  # Ignoring some data definition language errors
  slave-skip-errors=1452, 1062
  # Suppressing binary logs after a delay regardless of the replication status
  expire_logs_days=1
  # Maximum number of connections
  max_connections=1000
  # Size of each of the binary log files (default: 1GB)
  max_binlog_size=500M
  # Enabling writing to the DB in parallel threads for the replication
  slave-parallel-threads=10
  # enabling semi-synchronous replication
  rpl_semi_sync_master_enabled=ON
  rpl_semi_sync_slave_enabled=ON
  # change to READ COMMITTED
  transaction-isolation=READ-COMMITTED

• Restart the MariaDB servers on both hosts:

  sudo service mariadb restart

Semi-synchronous replication

By default, MariaDB uses asynchronous replication. To reach more consistency, it is recommended using semi-synchronous replication. The database configurations previously shown enable semi-synchronous replication with the following lines:

rpl_semi_sync_master_enabled=ON
rpl_semi_sync_slave_enabled=ON

info

MariaDB versions before 10.3.3 require the installation of plug-ins for semi-synchronous replication and are not supported.

Start-up of the replication

To start the replication, log in as root into your MariaDB client and run the following commands:

• on host-db1 (master is host-db2):

  CHANGE MASTER TO
  MASTER_HOST='host-db2',
  MASTER_USER='replication_user',
  MASTER_PASSWORD='replicationpassword',
  MASTER_PORT=3306,
  MASTER_USE_GTID=current_pos,
  MASTER_CONNECT_RETRY=10;

• on host-db2 (master is host-db1):

  CHANGE MASTER TO
  MASTER_HOST='host-db1',
  MASTER_USER='replication_user',
  MASTER_PASSWORD='replicationpassword',
  MASTER_PORT=3306,
  MASTER_USE_GTID=current_pos,
  MASTER_CONNECT_RETRY=10;

• on host-db1:

  START SLAVE;

• on host-db2:

  START SLAVE;

Additional setup

Purging the binary logs

With the provided configuration (expire_logs_days=1 in the MariaDB settings), the system will automatically remove the binary logs that are older than one day, even if the logs were not copied by the slave. This prevents the disk of the master node from being filled up in case the slave is down for a long time. The automatic binary log removal takes place when

• the master DB node starts,
• the logs are flushed (nevisAuth does not use this feature),
• the binary log rotates, or
• the binary logs are purged manually (see below).

So binary logs older than one day may exist, if none of the listed actions occurred recently.

Complementary to this expiration feature, MariaDB provides the possibility to manually purge the binary logs. The purge action removes all binary logs that were already copied by the slave. This allows a safe removal of the binary logs on a regular basis. The nevisProxy package is delivered with an adaptable purging script, which is located at: /opt/nevisproxy/sql/mariadb/purgebinarylogs.sh

To use this script,

• copy the script to a location of your choice, and
• adapt it to your configuration.

The script takes care of both DB nodes, so that it only needs to be configured once.

info

Note that if different database server nodes are used for nevisProxy and nevisAuth, you have set them up separately.

You can schedule the script to run for example once per hour, with a cron job:

0 * * * * /var/opt/nevisproxy/instance/conf/purgebinarylogs.sh # Absolute path of your adapted script

Size the binary logs

The provided configuration (max_binlog_size=500M in the MariaDB settings) allows you to configure the maximum size of the binary log files before rotating. The smaller the size, the more often rotations will occur, which will slow down replication. The advantage is a more efficient purge process. The bigger the size, the less often rotations will occur, but the disk may be filled with old logs.

According to our experiences, a size less than 8K does stop replication completely under heavy load, because the slave keeps rotating the logs.

Troubleshooting

Usually the slave stops the replication if an error occurs. You can check the state of the slave with the following SQL command:

show slave status\G

Note that showing the slave status requires the REPLICATION CLIENT grant.

If the replication has stopped, usually the error that caused it will be displayed. First you should try to fix the error. If this is not possible, you can do a "forced" restart of the slave like this:

• On the master call (to display the current state of the master):

  MariaDB [replicated_session_store]> show master status\G
  *************************** 1. row ***************************
   File: mariadbmaster-bin.000131
   Position: 194630804
   Binlog_Do_DB:
  Binlog_Ignore_DB:
  1 row in set (0.00 sec)

• On the slave (using the values returned by the call "show master status\G" on the master):

  STOP SLAVE;
  CHANGE MASTER TO
    MASTER_LOG_FILE='mariadbmaster-bin.000131',
    MASTER_LOG_POS=194630804;
  START SLAVE;

In this way, the system will restart the slave, without replicating to the slave all the sessions that occurred from the moment the replication has stopped until now.

Resilient database setup using PostgreSQL

Currently not supported.