Getting orientdb several exceptions when running a full suite of tests against a distributed cluster

Hello OrientDB team,
I’m trying a POC of distributed OrientDB (3.0.30) cluster of 3 nodes (master-master) and running a suite of 62 different tests,
each one creates a separate persistent database,
most of the tests are using live query.
every time I run the suite, after ~35 tests (which pass), I start to receive exceptions, usually the following ones:

    Suppressed: com.orientechnologies.orient.server.distributed.task.ODistributedOperationException: Request `2.21998` didn't reach the quorum of '2', responses: [node: odb1 success]
    DB name="Persistent_testObserveCount"
            at com.orientechnologies.orient.server.distributed.impl.ONewDistributedTransactionManager.handleResponse(ONewDistributedTransactionManager.java:329)
            at com.orientechnologies.orient.server.distributed.impl.ONewDistributedTransactionManager.retriedCommit(ONewDistributedTransactionManager.java:204)
            at com.orientechnologies.orient.server.distributed.impl.ONewDistributedTransactionManager.commit(ONewDistributedTransactionManager.java:84)
            at com.orientechnologies.orient.server.distributed.impl.ODatabaseDocumentDistributed.internalCommit(ODatabaseDocumentDistributed.java:346)
            at com.orientechnologies.orient.core.storage.impl.local.OMicroTransaction.doCommit(OMicroTransaction.java:363)
            at com.orientechnologies.orient.core.storage.impl.local.OMicroTransaction.commit(OMicroTransaction.java:166)
            at com.orientechnologies.orient.core.db.document.ODatabaseDocumentEmbedded.endMicroTransaction(ODatabaseDocumentEmbedded.java:798)
            at com.orientechnologies.orient.core.db.document.ODatabaseDocumentEmbedded.executeSaveRecord(ODatabaseDocumentEmbedded.java:788)
            at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveNew(OTransactionNoTx.java:244)
            at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveRecord(OTransactionNoTx.java:174)
            at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.saveInternal(ODatabaseDocumentAbstract.java:2077)
            at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.save(ODatabaseDocumentAbstract.java:2039)
            at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.save(ODatabaseDocumentAbstract.java:1929)
            at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.save(ODatabaseDocumentAbstract.java:86)
            at com.orientechnologies.orient.server.OConnectionBinaryExecutor.executeCreateRecord(OConnectionBinaryExecutor.java:390)
            at com.orientechnologies.orient.client.remote.message.OCreateRecordRequest.execute(OCreateRecordRequest.java:118)
            at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.sessionRequest(ONetworkProtocolBinary.java:310)
            at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolB

sometimes I get this exception:

10:08:21.547 [pool-2177-thread-1] DEBUG com.slimgears.rxrepo.orientdb.OrientDbSchemaProvider - Creating class: Manufacturer
10:08:21.686 [pool-2177-thread-1] DEBUG com.slimgears.rxrepo.query.DefaultEntitySet - Checking exception: Cannot set ownership of cluster ‘manufacturer_8’ to the server ‘odb2’, because the
server has no that cluster (sharding)
com.orientechnologies.orient.server.distributed.ODistributedException: Cannot set ownership of cluster ‘manufacturer_8’ to the server ‘odb2’, because the server has no that cluster (sharding)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.handleException(OChannelBinaryAsynchClient.java:338)
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.handleStatus(OChannelBinaryAsynchClient.java:285)
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.handleStatus(OChannelBinaryAsynchClient.java:307)
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:196)
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:158)
at com.orientechnologies.orient.client.remote.OStorageRemote.beginResponse(OStorageRemote.java:1888)
at com.orientechnologies.orient.client.remote.OStorageRemote.lambda$networkOperationRetryTimeout$2(OStorageRemote.java:245)
at com.orientechnologies.orient.client.remote.OStorageRemote.baseNetworkOperation(OStorageRemote.java:304)
at com.orientechnologies.orient.client.remote.OStorageRemote.networkOperationRetryTimeout(OStorageRemote.java:228)
at com.orientechnologies.orient.client.remote.OStorageRemote.networkOperationNoRetry(OStorageRemote.java:258)
at com.orientechnologies.orient.client.remote.OStorageRemote.command(OStorageRemote.java:944)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentRemote.command(ODatabaseDocumentRemote.java:407)
at com.orientechnologies.orient.core.metadata.schema.OSchemaRemote.createClass(OSchemaRemote.java:123)
at com.orientechnologies.orient.core.metadata.schema.OSchemaShared.createClass(OSchemaShared.java:205)
at com.orientechnologies.orient.core.metadata.schema.OSchemaProxy.createClass(OSchemaProxy.java:93)
at com.orientechnologies.orient.core.db.ODatabase.createClass(ODatabase.java:558)
at com.orientechnologies.orient.core.db.document.ODatabaseDocument.createClassIfNotExist(ODatabaseDocument.java:439)
at com.slimgears.rxrepo.orientdb.OrientDbSchemaProvider.createClass(OrientDbSchemaProvider.java:69)
at com.slimgears.rxrepo.orientdb.OrientDbSchemaProvider.lambda$null$3(OrientDbSchemaProvider.java:52)
at com.slimgears.rxrepo.orientdb.OrientDbSessionProvider.withSession(OrientDbSessionProvider.java:35)
at com.slimgears.rxrepo.orientdb.OrientDbSchemaProvider.lambda$createOrUpdate$4(OrientDbSchemaProvider.java:52)
at io.reactivex.internal.operators.completable.CompletableFromAction.subscribeActual(CompletableFromAction.java:35)
at io.reactivex.Completable.subscribe(Completable.java:2309)
at io.reactivex.internal.operators.completable.CompletableAndThenCompletable$SourceObserver.onComplete(CompletableAndThenCompletable.java:67)
at io.reactivex.internal.operators.completable.CompletableCache.onComplete(CompletableCache.java:88)
at io.reactivex.internal.operators.completable.CompletableCreate$Emitter.onComplete(CompletableCreate.java:64)
at com.slimgears.rxrepo.orientdb.OrientDbSchemaProvider.lambda$null$1(OrientDbSchemaProvider.java:40)
at com.slimgears.rxrepo.orientdb.OrientDbSessionProvider.lambda$withSession$2(OrientDbSessionProvider.java:43)
at com.slimgears.rxrepo.orientdb.OrientDbSessionProvider.withSession(OrientDbSessionProvider.java:35)
at com.slimgears.rxrepo.orientdb.OrientDbSessionProvider.withSession(OrientDbSessionProvider.java:42)
at com.slimgears.rxrepo.orientdb.OrientDbSchemaProvider.lambda$new$2(OrientDbSchemaProvider.java:34)
at io.reactivex.internal.operators.completable.CompletableCreate.subscribeActual(CompletableCreate.java:39)
at io.reactivex.Completable.subscribe(Completable.java:2309)
at io.reactivex.internal.operators.completable.CompletableCache.subscribeActual(CompletableCache.java:57)
at io.reactivex.Completable.subscribe(Completable.java:2309)
at io.reactivex.internal.operators.completable.CompletableAndThenCompletable.subscribeActual(CompletableAndThenCompletable.java:35)
at io.reactivex.Completable.subscribe(Completable.java:2309)
at io.reactivex.internal.operators.completable.CompletableAndThenCompletable$SourceObserver.onComplete(CompletableAndThenCompletable.java:67)
at io.reactivex.internal.operators.mixed.ObservableConcatMapCompletable$ConcatMapCompletableObserver.drain(ObservableConcatMapCompletable.java:258)
at io.reactivex.internal.operators.mixed.ObservableConcatMapCompletable$ConcatMapCompletableObserver.onComplete(ObservableConcatMapCompletable.java:162)
at io.reactivex.internal.observers.BasicFuseableObserver.onComplete(BasicFuseableObserver.java:119)
at io.reactivex.internal.observers.BasicFuseableObserver.onComplete(BasicFuseableObserver.java:119)
at io.reactivex.internal.operators.observable.ObservableFlatMapMaybe$FlatMapMaybeObserver.drainLoop(ObservableFlatMapMaybe.java:280)
at io.reactivex.internal.operators.observable.ObservableFlatMapMaybe$FlatMapMaybeObserver.drain(ObservableFlatMapMaybe.java:236)
at io.reactivex.internal.operators.observable.ObservableFlatMapMaybe$FlatMapMaybeObserver.onComplete(ObservableFlatMapMaybe.java:133)
:

followed by -

Caused by: com.orientechnologies.orient.core.exception.ODatabaseException: Cannot create database 'Persistent_testAddProductThenUpdateInventoryInOrderQueryAndObserve'
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.handleException(OChannelBinaryAsynchClient.java:338)
    at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.handleStatus(OChannelBinaryAsynchClient.java:285)
    at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.handleStatus(OChannelBinaryAsynchClient.java:307)
    at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:196)
    at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:158)
    at com.orientechnologies.orient.client.remote.OStorageRemote.beginResponse(OStorageRemote.java:1888)
    at com.orientechnologies.orient.client.remote.OServerAdmin$3.execute(OServerAdmin.java:446)
    at com.orientechnologies.orient.client.remote.OServerAdmin$3.execute(OServerAdmin.java:435)
    at com.orientechnologies.orient.client.remote.OServerAdmin.networkAdminOperation(OServerAdmin.java:472)
    ... 126 common frames omitted
    Suppressed: com.orientechnologies.orient.core.exception.ODatabaseException: Cannot create database 'Persistent_testAddProductThenUpdateInventoryInOrderQueryAndObserve'
            at com.orientechnologies.orient.core.db.OrientDBEmbedded.create(OrientDBEmbedded.java:391)
            at com.orientechnologies.orient.server.OServer.createDatabase(OServer.java:1236)
            at com.orientechnologies.orient.server.OConnectionBinaryExecutor.executeCreateDatabase(OConnectionBinaryExecutor.java:166)
            at com.orientechnologies.orient.client.remote.message.OCreateDatabaseRequest.execute(OCreateDatabaseRequest.java:95)
            at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.sessionRequest(ONetworkProtocolBinary.java:310)
            at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:212)
            at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:69)
    Caused by: com.orientechnologies.common.concur.lock.OLockException: Cannot release exclusive lock on resource 'Persistent_testAddProductThenUpdateInventoryInOrderQueryAndObserve'
            at com.orientechnologies.orient.server.distributed.impl.ODistributedLockManagerRequester.releaseExclusiveLock(ODistributedLockManagerRequester.java:171)
            at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.executeInDistributedDatabaseLock(ODistributedAbstractPlugin.java:1795)
            at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl.checkNodeInConfiguration(ODistributedDatabaseImpl.java:1114)
            at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.registerNewDatabaseIfNeeded(ODistributedAbstractPlugin.java:370)
            at com.orientechnologies.orient.server.hazelcast.ODistributedEmbeddedDatabaseInstanceFactory.newInstance(ODistributedEmbeddedDatabaseInstanceFactory.java:28)
            at com.orientechnologies.orient.core.db.OrientDBEmbedded.internalCreate(OrientDBEmbedded.java:450)
            at com.orientechnologies.orient.core.db.OrientDBEmbedded.create(OrientDBEmbedded.java:389)
            ... 6 common frames omitted
Caused by: com.orientechnologies.common.concur.lock.OLockException: Cannot release exclusive lock on resource 'Persistent_testAddProductThenUpdateInventoryInOrderQueryAndObserve'
    at com.orientechnologies.orient.server.distributed.impl.ODistributedLockManagerRequester.releaseExclusiveLock(ODistributedLockManagerRequester.java:171)
    at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.executeInDistributedDatabaseLock(ODistributedAbstractPlugin.java:1795)
    at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl.checkNodeInConfiguration(ODistributedDatabaseImpl.java:1114)
    at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.registerNewDatabaseIfNeeded(ODistributedAbstractPlugin.java:370)
    at com.orientechnologies.orient.server.hazelcast.ODistributedEmbeddedDatabaseInstanceFactory.newInstance(ODistributedEmbeddedDatabaseInstanceFactory.java:28)
    at com.orientechnologies.orient.core.db.OrientDBEmbedded.internalCreate(OrientDBEmbedded.java:450)
    at com.orientechnologies.orient.core.db.OrientDBEmbedded.create(OrientDBEmbedded.java:389)
    at com.orientechnologies.orient.server.OServer.createDatabase(OServer.java:1236)
    at com.orientechnologies.orient.server.OConnectionBinaryExecutor.executeCreateDatabase(OConnectionBinaryExecutor.java:166)
    at com.orientechnologies.orient.client.remote.message.OCreateDatabaseRequest.execute(OCreateDatabaseRequest.java:95)
    at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.sessionRequest(ONetworkProtocolBinary.java:310)
    at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:212)
    at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:69)

and

Caused by: com.orientechnologies.common.concur.lock.OLockException: Server 'odb1' cannot acquire exclusive lock on resource 'Persistent_testAtomicUpdate' (timeout=20000)

and after that, all the tests fail because they can’t create their databases because of this error (cannot accquire exclusive lock)

please advise -
what could be causing that?
any tips on how to proceed?

this is the configuration:

{
“autoDeploy”: true,
“readQuorum”: 1,
“writeQuorum”: “majority”,
“executionMode”:“undefined”,
“readYourWrites”: true,
“newNodeStrategy”: “dynamic”,
“servers”: {
": “master”
},
“clusters”: {
“internal”: {
},
"
”: {
“servers”: ["<NEW_NODE>"]
}
}
}

this is the compose file -

version: ‘3.3’
services:
odb1:
image: orientdb:3.0.30
command: dserver.sh
volumes:
- ./var/odb1/config:/orientdb/config
- ./var/odb1/databases:/orientdb/databases
environment:
ORIENTDB_ROOT_PASSWORD: root
ORIENTDB_NODE_NAME: odb1
ORIENTDB_OPTS_MEMORY: -Xms4G -Xmx4G
ports:
- 2480:2480
- 2424:2424

odb2:
image: orientdb:3.0.30
command: dserver.sh
volumes:
- ./var/odb2/config:/orientdb/config
- ./var/odb2/databases:/orientdb/databases
environment:
ORIENTDB_ROOT_USER: root
ORIENTDB_NODE_NAME: odb2
ORIENTDB_OPTS_MEMORY: -Xms4G -Xmx4G
depends_on:
- odb1

odb3:
image: orientdb:3.0.30
command: dserver.sh
volumes:
- ./var/odb3/config:/orientdb/config
- ./var/odb3/databases:/orientdb/databases
environment:
ORIENTDB_ROOT_PASSWORD: root
ORIENTDB_NODE_NAME: odb3
ORIENTDB_OPTS_MEMORY: -Xms4G -Xmx4G
depends_on:
- odb2

Edit:
I also noticed that I get those warnings regarding open files limit.
I increased the limit on the Linux VM where the nodes are running to 500000,
however, I keep getting those warnings, is it a parameter that needs to be passed to the dserver.sh inside the docker-compse.yml? or to the docker container? if so how to do it?

com.slimgears.rxrepo.orientdb.RemoteOrientDbQueryProviderTest > testAddProductThenUpdateInventoryInOrderQueryAndObserve STANDARD_ERROR
Jun 08, 2020 9:16:49 AM com.orientechnologies.common.log.OLogManager log
INFO: Detected limit of amount of simultaneously open files is 4096, limit of open files for disk cache will be set to 1536
Jun 08, 2020 9:16:49 AM com.orientechnologies.common.log.OLogManager log
WARNING: Value of limit of simultaneously open files is too small, recommended value is 262144

let me know if you need more information,
Thanks,
Anat Bulanov.