Couchbase Inconsistency Repair Tool

Introduction

In versions of nevisMeta before 1.1.0.0, importing and deleting setups or entities could lead to inconsistencies in the persisted setup data. Such inconsistencies include entities with references to other non-existent entities or entity states. This significantly slows down the system and leads to log entries such as the following:

2018-02-21 14:01:58,767 rsistenceServiceBean:236  [        ] (http-thread-pool-8996(1)) INFO  Failed to load Object with id 'Client_ca8fb0095b179a4213cebd84c291a7a2', no such document in store

To fix inconsistencies caused in earlier versions of nevisMeta, a repair tool is provided. The repair tool fixes the following inconsistencies:

• It removes references to entities and states which cannot be reached.
• It removes states or entities that are not referenced.

Usage

Backup

We strongly advise to always make a backup of a Couchbase bucket, before you use the Couchbase Inconsistency Repair Tool or perform any other direct manipulation of data in Couchbase. See /opt/couchbase/bin/cbbackup --help for help on how to create a backup of a Couchbase bucket.

The Couchbase Inconsistency Repair Tool is available after installing nevisMeta, under /opt/nevismeta/tools/nevismeta-cb-repair.jar.

Running nevismeta-cb-repair.jar without arguments (or with "-h" for help), will display the usage help:

java -jar nevismeta-cb-repair.jar

The usage help lists all available parameters including an example of how to use the repair tool:

'nevismeta-cb-repair' may be used to repair inconsistent data in the Couchbase database.
 
usage: NevisMeta Couchbase repair
 -b,--bucket <arg>        Bucket name to repair.
 -h,--help                Show usage.
 -host,--hostname <arg>   The name of the host the Couchbase server is
                          running on.
 -p,--protocol <arg>      The protocol 'http' or 'https'. Default value is
                          'http'.
 -po,--port <arg>         The port the Couchbase server is running on.
                          Default value is '8091'.
 -pw,--password           Prompt for the password to connect to the
                          bucket.
 
example:
$ java -jar nevismeta-cb-repair.jar -host=some.hostname.com -b=bucketName -pw -po=12345 -p=https

Mandatory parameters are: hostname and bucket.

Troubleshooting

Below find troubleshooting for the following cases:

• Wrong hostname or bucket
• Password does not match

Wrong hostname or bucket

When the provided hostname or bucket are wrong, the tool will typically throw a TimeoutException. In this case, double check the hostname or bucket name.

2018-08-28 11:14:24,505 INFO  [main] [tamodel.couchbase.CouchbaseDocumentStore:16  ] nodes: some.hostname.com
2018-08-28 11:14:24,509 INFO  [main] [tamodel.couchbase.CouchbaseDocumentStore:17  ] bucket: bucketName
2018-08-28 11:14:24,509 INFO  [main] [tamodel.couchbase.CouchbaseDocumentStore:18  ] password:
2018-08-28 11:14:24,761 INFO  [main] [ com.couchbase.client.core.CouchbaseCore:144 ] CouchbaseEnvironment: {sslEnabled=false, sslKeystoreFile='null', sslKeystorePassword='null', queryEnabled=false, queryPort=8093, bootstrapHttpEnabled=true, bootstrapCarrierEnabled=true, bootstrapHttpDirectPort=8091, bootstrapHttpSslPort=18091, bootstrapCarrierDirectPort=11210, bootstrapCarrierSslPort=11207, ioPoolSize=4, computationPoolSize=4, responseBufferSize=16384, requestBufferSize=16384, kvServiceEndpoints=1, viewServiceEndpoints=1, queryServiceEndpoints=1, searchServiceEndpoints=1, ioPool=NioEventLoopGroup, coreScheduler=CoreScheduler, eventBus=DefaultEventBus, packageNameAndVersion=couchbase-java-client/2.2.8 (git: 2.2.8, core: 1.2.9), dcpEnabled=false, retryStrategy=BestEffort, maxRequestLifetime=75000, retryDelay=ExponentialDelay{growBy 1.0 MICROSECONDS, powers of 2; lower=100, upper=100000}, reconnectDelay=ExponentialDelay{growBy 1.0 MILLISECONDS, powers of 2; lower=32, upper=4096}, observeIntervalDelay=ExponentialDelay{growBy 1.0 MICROSECONDS, powers of 2; lower=10, upper=100000}, keepAliveInterval=30000, autoreleaseAfter=2000, bufferPoolingEnabled=true, tcpNodelayEnabled=true, mutationTokensEnabled=false, socketConnectTimeout=1000, dcpConnectionBufferSize=20971520, dcpConnectionBufferAckThreshold=0.2, dcpConnectionName=dcp/core-io, callbacksOnIoPool=false, queryTimeout=75000, viewTimeout=75000, kvTimeout=2500, connectTimeout=5000, disconnectTimeout=25000, dnsSrvEnabled=false}
2018-08-28 11:14:27,193 INFO  [main] [tamodel.couchbase.CouchbaseDocumentStore:26  ] New CouchbaseCluster: com.couchbase.client.java.CouchbaseCluster@7d0587f1
2018-08-28 11:14:32,214 ERROR [main] [tamodel.couchbase.CouchbaseDocumentStore:140 ] Failed to open bucket 'bucketName': java.util.concurrent.TimeoutException
2018-08-28 11:14:37,416 ERROR [main] [tamodel.couchbase.CouchbaseDocumentStore:140 ] Failed to open bucket 'bucketName': java.util.concurrent.TimeoutException
2018-08-28 11:14:42,618 ERROR [main] [tamodel.couchbase.CouchbaseDocumentStore:140 ] Failed to open bucket 'bucketName': java.util.concurrent.TimeoutException
2018-08-28 11:14:47,820 ERROR [main] [tamodel.couchbase.CouchbaseDocumentStore:140 ] Failed to open bucket 'bucketName': java.util.concurrent.TimeoutException
2018-08-28 11:14:53,023 ERROR [main] [tamodel.couchbase.CouchbaseDocumentStore:140 ] Failed to open bucket 'bucketName': java.util.concurrent.TimeoutException
Exception in thread "main" java.lang.RuntimeException: Failed to create Couchbase bucket: java.lang.RuntimeException: java.util.concurrent.TimeoutException
    at ch.nevis.nevismeta.datamodel.couchbase.CouchbaseDocumentStore.initBucket(CouchbaseDocumentStore.java:149)
    at ch.nevis.nevismeta.datamodel.couchbase.CouchbaseDocumentStoreImpl.init(CouchbaseDocumentStoreImpl.java:34)
    at ch.nevis.nevismeta.datamodel.couchbase.CouchbaseDocumentStore.init(CouchbaseDocumentStore.java:81)
    at ch.nevis.nevismeta.NevisMetaCouchbaseRepair.main(NevisMetaCouchbaseRepair.java:30)
Caused by: java.lang.RuntimeException: java.util.concurrent.TimeoutException
    at com.couchbase.client.java.util.Blocking.blockForSingle(Blocking.java:75)
    at com.couchbase.client.java.CouchbaseCluster.openBucket(CouchbaseCluster.java:296)
    at com.couchbase.client.java.CouchbaseCluster.openBucket(CouchbaseCluster.java:277)
    at com.couchbase.client.java.CouchbaseCluster.openBucket(CouchbaseCluster.java:266)
    at ch.nevis.nevismeta.datamodel.couchbase.CouchbaseDocumentStore.initBucket(CouchbaseDocumentStore.java:132)
    ... 3 more
Caused by: java.util.concurrent.TimeoutException
    ... 8 more

Password does not match

An invalid password will lead to an InvalidPasswordException.

2018-08-28 11:20:23,320 INFO  [main] [tamodel.couchbase.CouchbaseDocumentStore:16  ] nodes: nevismeta-test5.adnovum.vn
2018-08-28 11:20:23,323 INFO  [main] [tamodel.couchbase.CouchbaseDocumentStore:17  ] bucket: default
2018-08-28 11:20:23,323 INFO  [main] [tamodel.couchbase.CouchbaseDocumentStore:18  ] password: ********
2018-08-28 11:20:23,513 INFO  [main] [ com.couchbase.client.core.CouchbaseCore:144 ] CouchbaseEnvironment: {sslEnabled=false, sslKeystoreFile='null', sslKeystorePassword='null', queryEnabled=false, queryPort=8093, bootstrapHttpEnabled=true, bootstrapCarrierEnabled=true, bootstrapHttpDirectPort=8091, bootstrapHttpSslPort=18091, bootstrapCarrierDirectPort=11210, bootstrapCarrierSslPort=11207, ioPoolSize=4, computationPoolSize=4, responseBufferSize=16384, requestBufferSize=16384, kvServiceEndpoints=1, viewServiceEndpoints=1, queryServiceEndpoints=1, searchServiceEndpoints=1, ioPool=NioEventLoopGroup, coreScheduler=CoreScheduler, eventBus=DefaultEventBus, packageNameAndVersion=couchbase-java-client/2.2.8 (git: 2.2.8, core: 1.2.9), dcpEnabled=false, retryStrategy=BestEffort, maxRequestLifetime=75000, retryDelay=ExponentialDelay{growBy 1.0 MICROSECONDS, powers of 2; lower=100, upper=100000}, reconnectDelay=ExponentialDelay{growBy 1.0 MILLISECONDS, powers of 2; lower=32, upper=4096}, observeIntervalDelay=ExponentialDelay{growBy 1.0 MICROSECONDS, powers of 2; lower=10, upper=100000}, keepAliveInterval=30000, autoreleaseAfter=2000, bufferPoolingEnabled=true, tcpNodelayEnabled=true, mutationTokensEnabled=false, socketConnectTimeout=1000, dcpConnectionBufferSize=20971520, dcpConnectionBufferAckThreshold=0.2, dcpConnectionName=dcp/core-io, callbacksOnIoPool=false, queryTimeout=75000, viewTimeout=75000, kvTimeout=2500, connectTimeout=5000, disconnectTimeout=25000, dnsSrvEnabled=false}
2018-08-28 11:20:23,600 INFO  [main] [tamodel.couchbase.CouchbaseDocumentStore:26  ] New CouchbaseCluster: com.couchbase.client.java.CouchbaseCluster@7d0587f1
2018-08-28 11:20:23,955 WARN  [cb-io-1-1] [.couchbase.client.core.endpoint.Endpoint:292 ] [null][KeyValueEndpoint]: Authentication Failure.
2018-08-28 11:20:25,108 WARN  [cb-io-1-3] [nt.core.endpoint.ResponseStatusConverter:151 ] Unknown ResponseStatus with Protocol HTTP: 401
Exception in thread "main" java.lang.RuntimeException: Failed to create Couchbase bucket: com.couchbase.client.java.error.InvalidPasswordException: Passwords for bucket "default" do not match.
    at ch.nevis.nevismeta.datamodel.couchbase.CouchbaseDocumentStore.initBucket(CouchbaseDocumentStore.java:135)
    at ch.nevis.nevismeta.datamodel.couchbase.CouchbaseDocumentStoreImpl.init(CouchbaseDocumentStoreImpl.java:34)
    at ch.nevis.nevismeta.datamodel.couchbase.CouchbaseDocumentStore.init(CouchbaseDocumentStore.java:81)
    at ch.nevis.nevismeta.NevisMetaCouchbaseRepair.main(NevisMetaCouchbaseRepair.java:30)
Caused by: com.couchbase.client.java.error.InvalidPasswordException: Passwords for bucket "default" do not match.
    at com.couchbase.client.java.CouchbaseAsyncCluster$OpenBucketErrorHandler.call(CouchbaseAsyncCluster.java:480)
    at com.couchbase.client.java.CouchbaseAsyncCluster$OpenBucketErrorHandler.call(CouchbaseAsyncCluster.java:463)
    at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$1.onError(OperatorOnErrorResumeNextViaFunction.java:99)
    at rx.internal.operators.OperatorMap$1.onError(OperatorMap.java:48)
    at rx.observers.Subscribers$5.onError(Subscribers.java:229)
    at rx.internal.operators.OperatorObserveOn$ObserveOnSubscriber.pollQueue(OperatorObserveOn.java:191)
    at rx.internal.operators.OperatorObserveOn$ObserveOnSubscriber$2.call(OperatorObserveOn.java:162)
    at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)