Skip to main content
Version: 1.5.x.x LTS

Migrating from Couchbase to MariaDB

This guide describes the migration from Couchbase to MariaDB as the persistence backend.

We strongly recommend testing the migration in a test environment to ensure that all functional and performance requirements are met.

Prerequisites​

  • You need a nevisMeta version of >= 1.5.3.x
  • The MariaDB database must be created and empty, see Creating a new database for nevisMeta.

Procedure​

The basic process consists of the following steps:

  1. Ensure the prerequisites are met.
  2. Create a backup of Couchbase and MariaDB.
  3. Execute the data migration with the provided migration tool.
  4. Execute tests to ensure that the migration was successfully and that the new setup meets your performance and functional requirements.

Backup​

  • Back up Couchbase and MariaDB before the migration as described in Backup.

Data migration​

Use the provided nevisMeta Migration Tool for the migration to MariaDB. This tool is available after installing nevisMeta, under /opt/nevismeta/tools/nevismeta-couchbase-to-mariadb.jar.

Perform these steps​

  1. Run the command nevismeta-couchbase-to-mariadb.jar without arguments (or with "-h" for help):
java -jar nevismeta-couchbase-to-mariadb.jar

This will display the usage help of the migration tool, as you can see in the next code block. The usage help lists all available parameters including an example of how to use the migration tool:

'nevismeta-couchbase-to-mariadb' is used to migrate data from Couchbase to MariaDb.
 
usage: nevisMeta Migration tool
 -c,--migrateUserConsent <arg>    Migrate user consents. Default is
                                  'false' which will not migrate persisted
                                  user consents.
 -d,--destinationUrl <arg>        The destination URL of MariaDB.
 -h,--help                        Show usage.
 -l,--logLevel <arg>              Change log level. Supported value:
                                  DEBUG, INFO, WARN, ERROR. Default is
                                  'INFO'.
 -p,--password <arg>              Password to connect to MariaDB. Skip to
                                  prompt for input.
 -r,--removeData <arg>            Remove all data in database.  Default is
                                  'false' which will not remove data in
                                  database.
 -s,--sourceUrl <arg>             The source URL of Couchbase.
 -t,--migrateRefreshToken <arg>   Migrate refresh tokens. Default is
                                  'false' which will not migrate refresh
                                  tokens.
 -u,--username <arg>              Username to connect to MariaDB.
 
example:
$ java -jar nevismeta-couchbase-to-mariadb.jar -s=couchbase:http://your-couchbase-host.com:8091/bucket_name?password=password -d=jdbc:mariadb://your-mariadb-host.com:3306/nevismeta-db-name -u=username
  1. Migrate the data from Couchbase to MariaDB with the nevisMeta Migration Tool. See the usage help above for an overview of the available arguments.
  2. The migration process copies all data from Couchbase to MariaDB. Wait for the process to complete.

Testing​

Execute tests to ensure that

  • the data migration was successful and correct,
  • the system is fully functional, and
  • the system meets the performance requirements.

To get a first assessment of the situation after the migration, do the following:

  1. Check that all setups have been migrated.
  2. Randomly select a few clients within a setup. Ensure that the following elements are available:
  • The scopes for which the client is registered.
  • The expected meta information about the client.
  • The expected client states.
  1. Randomly select a few resource servers. Check that
  • the expected scopes in the resource server exist,
  • the expected meta information on the resource server exists, and
  • all expected resource server states are there.

Performance​

The migration tool migrates the entities in batches of 10'000 pieces. The list below shows the time needed for the migration in a test environment - as an indication for the migration in the production environment:

  • Time to migrate the resource server:
    • 21 resource servers with 105 scopes in total: 2s
    • 52 resource servers with 65 scopes in total: 2s
    • 2'000 resource servers with 10'000 scopes in total: 48s
  • Time to migrate the client:
    • 24 clients: 2s
    • 308 clients: 8s
    • 1'878 clients: 1m 12s
  • Average time to migrate 10'000 refresh tokens: 10s
    • The time needed to migrate all refresh tokens is linear with the total number of refresh tokens, in batches of 10'000 pieces.
  • Average time to migrate 10'000 persisted consents: 10s
    • The time needed to migrate all persisted consents is linear with the total number of persisted consents, in batches of 10'000 pieces.

Removing data before migration​

The time that you need to remove all data from a database before the migration depends on the size of the database you want to delete. For example, removing 7 million records from a database takes about 12 minutes.

Troubleshooting​

This section discusses problems that you can encounter when migrating from Couchbase to MariaDB.

Problem: Duplicate entity name​

Message example​

2019-11-25 15:01:54,917 WARN  Setup with id '370ca792a0bd3476c91bd78cdc65fa16' contain a duplicated name 'Setup'. Changed name to 'Setup_2'

Reason: To avoid misconfigurations, the following fields in MariaDB contain unique constraints:

  • Setup name.
  • Client name within a setup.
  • Client ID within a setup.
  • Resource server name within a setup.

How to fix​

  • You do not need to perform a manual fix. The nevisMeta Migration Tool automatically adds a suffix number to migrated entities with the same name. For example, if the tool imports a setup named "MySetup", and there already exists another setup with the same name, then the tool automatically renames the imported setup to "MySetup_2".

Problem: Duplicate entity ID​

Message example​

2019-11-25 15:15:52,818 ERROR Insert to Setup fail.
java.sql.BatchUpdateException: (conn=6624) Duplicate entry 'H\x8D\x04ub\xB3\xAFd\xF0\xB7|\xABo\x0E\x95\xC4' for key 'uq_setup_resource_id'
    at org.mariadb.jdbc.MariaDbStatement.executeBatchExceptionEpilogue(MariaDbStatement.java:282)
    at org.mariadb.jdbc.MariaDbPreparedStatementClient.executeBatch(MariaDbPreparedStatementClient.java:294)
    at com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:128)
    at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java)
    at ch.nevis.nevismeta.service.BatchExecutionUtils.executePreparedStatement(BatchExecutionUtils.java:34)
    at ch.nevis.nevismeta.service.SetupMigrator.insertBatchSetups(SetupMigrator.java:100)
    at ch.nevis.nevismeta.service.SetupMigrator.migrateSetup(SetupMigrator.java:159)
    at ch.nevis.nevismeta.service.SetupMigrator.migrate(SetupMigrator.java:132)
    at ch.nevis.nevismeta.MigratingToSqlDatabaseTypeApplication.migrateData(MigratingToSqlDatabaseTypeApplication.java:82)
    at ch.nevis.nevismeta.MigratingToSqlDatabaseTypeApplication.main(MigratingToSqlDatabaseTypeApplication.java:72)

Reason: The imported setup has the same ID as an existing setup.

How to fix:

  • Add the following argument to the migration tool: -r=true. This will remove all data in MariaDB.

Problem: Entity value is too long​

Error message example​

Caused by: java.sql.SQLException: Data too long for column 'name' at row 1
Query is: insert into tmetc_client (ctl_tcn, ctl_cre_uid, ctl_cre_ts, ctl_mod_uid, ctl_mod_ts, name, owner, resource_id, setup_id, oauth2_client_id)values (0,'migration',CURRENT_TIMESTAMP(),'migration',CURRENT_TIMESTAMP(),'xxx1D7hMOhWDpgz3Lot0OcqRA8NWsYGbzw2ZOhUXGFpexSDQBUzkVGVGn1K7ymJGzDy0dvYsmw7MAM8bpciUk2ZUqE6qTDO6HqdbEqUfExYlzf2VknunyLSChl8ClM3iSycfeMipd7NRJAMwM5tu6ZDHmFq4ui81KhzZbY2KhqXgywiBGwe7kDovUNCZRGDmJTcegmrOH32Sr0j0TLlj8WFYgxfbwii54NFsRynnzwfTWAgPTWSmNGsdGM1ejxPRVPLjY3v3jlRaTFAjNFSds6fXkbr3MGf2nDvc4BXuUzc0JSm2TvSKmMOZ9q5AOkYBAoxCqUzUz1zbaqQvtN0wWj0rHMmZBQ3NVweqNV9XkT5vJL4hTfAKYt6LpsozxkhFQGvixhNVvvlIgLCnEcHPYL1OpDWKoXq6OyIU71jEW67szj6Vxpp97bD2LyCcHJhcUQ4d0RahVr4runduNXGHblKIRDoTWYU75IVzFCrnPNuWg3OrvJSjwRrcKDDFCuW9dLBP','vvv',<bytearray:���B_��+H�9�Q>, 3, 'xxx1');
        at org.mariadb.jdbc.internal.util.LogQueryTool.exceptionWithQuery(LogQueryTool.java:113)
        at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol$1.handleResultException(AbstractQueryProtocol.java:533)
        at org.mariadb.jdbc.internal.protocol.AsyncMultiRead.call(AsyncMultiRead.java:139)
        at org.mariadb.jdbc.internal.protocol.AsyncMultiRead.call(AsyncMultiRead.java:67)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Reason: The maximum length of some fields in MariaDB differs from the fields' maximum length in Couchbase. The list below shows these fields, including the new maximum lengths:

  • Setup:
    • name: 255 characters maximum.
    • owner, editor: 500 characters maximum.
    • comment: 65'535 characters maximum.
  • Client:
    • name, owner, oauth2_client_id, editor, oauth2_client_secret: 500 characters maximum.
    • redirect_uris, contacts, initiate_login_uri, comment: 65'535 characters maximum
  • Resource Server:
  • name, owner, editor: 500 characters maximum.
  • uri, comment: 65,535 characters maximum.
  • Scope:
    • name: 500 characters maximum.
  • Refresh Token:
    • value, oauth2_client_id, user, issuer: 500 characters maximum.
    • claims, custom_access_token_claims, custom_id_token_claims: 4000 characters maximum.
  • Persisted Consent:
    • oauth2_client_id, user: 500 characters maximum.

How to fix: Before the migration, change the values of the affected fields in Couchbase to values shorter than the allowed maximum length in MariaDB. You can do this directly via the Couchbase web interface or via an existing instance configured to use Couchbase as the persistence backend. Restart the migration after changing the affected field values.

Problem: Scope in client cannot be migrated​

Message example​

2019-11-25 15:07:11,336 WARN  Skip scope '[value=authorization_scope, resource server id = ResourceServer_a6764a65dd0263031f4925b18139f08c]' using by Client with id 'Client_71d502b44204fbf5a8d7991a1d02a671' because it is associated with an not-existed resource server.

Reason: To avoid inconsistency, the migration tool will not migrate a scope in a client that is associated with a deleted resource server.

Problem: Refresh token cannot be migrated​

Message example​

2019-11-25 15:10:09,813 WARN  Skip the refresh token with value of 'token-b' because it does not associate with any valid client.

Reason: To avoid inconsistency, the migration tool will only migrate refresh tokens that are associated with a valid client.

Message example​

2019-11-25 17:33:44,850 WARN  Skip the refresh token with value of 'token-b' because its client identifier 'c1i' is not sufficient to determine an associated client.

Reason: Many clients have the same oauth client identifier. Therefore, the migration tool ignores all tokens for which it cannot determine the associated client.

Message example​

2019-11-25 15:12:25,324 WARN Skip Persisted Consent OAuth2PersistedConsentEntity because it does not associate with any valid client.

Reason: To avoid inconsistency, the migration tool will only migrate persistent consents that are associated with a valid client.

Problem: DB: PROCEDURE xxx does not exist (SQL State: null - Error Code: 1305)​

Message example​

2019-11-25 15:12:25,324 WARN  DB: PROCEDURE nevismeta.update_column_name_refresh_token_ttl does not exist (SQL State: null - Error Code: 1305)
2019-11-25 15:12:25,324 WARN  DB: PROCEDURE nevismeta.update_unique_constrain_for_nevismeta_1_5_3 does not exist (SQL State: null - Error Code: 1305)

Reason: Some internally stored procedures of nevisMeta are only created if they do not exist. If they are already created during the migration, a warning is written. If this message appears as a warning (preceded by the word "WARN" in the log line), it is normal and may be safely ignored.

Problem: DB: Name 'flyway_schema_history_pk' ignored for PRIMARY key. (SQL State: 42000 - Error Code: 1280)​

Message example​

2019-11-25 15:12:25,324 WARN  DB: Name 'flyway_schema_history_pk' ignored for PRIMARY key. (SQL State: 42000 - Error Code: 1280)

Reason: An internal primary key of nevisMeta is only created if it does not exist. If it is already created at the moment of migrating, a warning is written. If this message appears as a warning (preceded by the word "WARN" in the log line), it is normal and may be safely ignored.

Problem: ResourceServer with id 'xxx' contains a duplicated name 'nnn'. Changed name to 'mmm'​

Message example​

2019-11-25 15:12:25,324 WARN  ResourceServer with id 'ResourceServer_7405fc41cb9ey4b8ec7ed3437b30631d' contain a duplicated name 'myserver'. Changed name to 'myserver_2'

Reason: nevisMeta sanitizes the original data of the Couchbase database before writing it in the MariaDB database. One of these sanitizing measures is to avoid having two resource server names in the same setup with the same name. If nevisMeta detects this situation it will rename one of the resource servers with a prefix.

Note: If the original data from Couchbase has some inconsistencies in its structure, it may confuse nevisMeta and it will do the renaming even with the duplicated resource servers in different setups. The workaround to this is simply to rename the resource server after the migration to its original name.