Nodes fail to merge in distributed mode

Hello,

Lately, I’ve encountered several different issues where two or more nodes are not able to join back together after one has been shut down. We are running a setup of one master and two replicas on 3.0.21 with default Hazelcast configurations and default OrientDB configs (except for one or two settings).

The simplest of the issues is when the following message is printed, but then nothing happens [10.200.114.244]:2434 [orientdb] [3.10.6] [10.200.114.246]:2434 should merge to us, both have the same data member count: 1 [ClusterJoinManager]. This (obviously?) only occurs if there is only one alive node. What is the reason why the nodes are acknowledging that they should merge to each other but then, in the end, don’t? This issue doesn’t always reproduce if restarting one of the involved nodes.

However, the issue that is bugging us more is when the nodes initiate a merge procedure, but then it fails as seen in the below log snippet. This procedure repeats itself every ~5min. To my untrained eye, it looks as if the merging node is shut down prematurely, but it is not clear what the reason for that is. It always reproduces and the only solution to get out of this problem that I found is to restart both nodes. But when starting the third node this issue also appears and I haven’t found any solution to resolve it.

Any help with this issue is appreciated!

/ Sebastian

+--------+-------+------------------------------------------+-----+---------+-------------------+-------------------+-----------------------+
|Name    |Status |Databases                                 |Conns|StartedOn|Binary             |HTTP               |UsedMemory             |
+--------+-------+------------------------------------------+-----+---------+-------------------+-------------------+-----------------------+
|01901(*)|MERGING|ebrslxn-allevents=NOT_AVAILABLE (REPLICA) |0    |09:37:47 |10.200.114.245:2424|10.200.114.245:2480|1.02GB/10.67GB (9.60%) |
|        |       |ebrslxn-allevents2=NOT_AVAILABLE (REPLICA)|     |         |                   |                   |                       |
|        |       |esebtor-allevents=NOT_AVAILABLE (REPLICA) |     |         |                   |                   |                       |
|        |       |allevents=NOT_AVAILABLE (REPLICA)         |     |         |                   |                   |                       |
|01900(@)|ONLINE |ebrslxn-allevents=ONLINE (MASTER)         |4    |09:22:12 |10.200.114.244:2424|10.200.114.244:2480|1.12GB/10.67GB (10.49%)|
|        |       |ebrslxn-allevents2=NOT_AVAILABLE (MASTER) |     |         |                   |                   |                       |
|        |       |esebtor-allevents=NOT_AVAILABLE (MASTER)  |     |         |                   |                   |                       |
|        |       |allevents=NOT_AVAILABLE (MASTER)          |     |         |                   |                   |                       |
+--------+-------+------------------------------------------+-----+---------+-------------------+-------------------+-----------------------+
 [OHazelcastPlugin]
2020-04-14 09:42:59:438 WARNI [01901]<-[01900] Restarting server... [ORestartServerTask]
2020-04-14 09:42:59:438 INFO  [01901] Updated node status to 'OFFLINE' [OHazelcastPlugin]
2020-04-14 09:42:59:440 INFO  OrientDB Server is shutting down... [OServer]
2020-04-14 09:42:59:440 INFO  Shutting down listeners: [OServer]
2020-04-14 09:42:59:440 INFO  - ONetworkProtocolBinary /0.0.0.0:2424: [OServer]
2020-04-14 09:42:59:441 INFO  - ONetworkProtocolHttpDb /0.0.0.0:2480: [OServer]
2020-04-14 09:42:59:442 INFO  Shutting down protocols [OServer]
2020-04-14 09:42:59:445 INFO  Shutting down plugins: [OServerPluginManager]
2020-04-14 09:42:59:445 INFO  - studio [OServerPluginManager]
2020-04-14 09:42:59:445 INFO  - cluster [OServerPluginManager]
2020-04-14 09:42:59:445 WARNI Shutting down node '01901'... [OHazelcastPlugin]
2020-04-14 09:42:59:445 INFO  [01901] Updated node status to 'SHUTTINGDOWN' [OHazelcastPlugin]
2020-04-14 09:42:59:657 INFO  [01901] Shutting down distributed database manager 'ebrslxn-allevents'. Pending objects: txs=0 locks=0 [ODistributedDatabaseImpl]
2020-04-14 09:42:59:935 INFO  [01901] Shutting down distributed database manager 'ebrslxn-allevents2'. Pending objects: txs=0 locks=0 [ODistributedDatabaseImpl]
2020-04-14 09:43:00:146 INFO  [01901] Shutting down distributed database manager 'esebtor-allevents'. Pending objects: txs=0 locks=0 [ODistributedDatabaseImpl]
2020-04-14 09:43:00:238 INFO  [01901] Shutting down distributed database manager 'allevents'. Pending objects: txs=0 locks=0 [ODistributedDatabaseImpl]
2020-04-14 09:43:00:238 INFO  [01901] Updated node status to 'OFFLINE' [OHazelcastPlugin]
2020-04-14 09:43:00:238 WARNI [01901] Network merged, lockManager=01900... [OHazelcastPlugin$5][10.200.114.245]:2434 [orientdb] [3.10.6] hz._hzInstance_1_orientdb.event-1 caught an exception while processing task:com.hazelcast.spi.impl.eventservice.impl.LocalEventDispatcher@51696e3f
com.orientechnologies.orient.server.distributed.task.ODistributedOperationException: Request has been canceled
	at com.orientechnologies.orient.server.distributed.ODistributedResponseManagerImpl.waitForSynchronousResponses(ODistributedResponseManagerImpl.java:283)
	at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl.waitForResponse(ODistributedDatabaseImpl.java:1097)
	at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl.send2Nodes(ODistributedDatabaseImpl.java:549)
	at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl.send2Nodes(ODistributedDatabaseImpl.java:420)
	at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.sendRequest(ODistributedAbstractPlugin.java:630)
	at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.sendRequest(ODistributedAbstractPlugin.java:592)
	at com.orientechnologies.orient.server.distributed.impl.ODistributedLockManagerRequester.acquireExclusiveLock(ODistributedLockManagerRequester.java:78)
	at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.executeInDistributedDatabaseLock(ODistributedAbstractPlugin.java:1756)
	at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.installDatabase(ODistributedAbstractPlugin.java:947)
	at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.entryUpdated(OHazelcastPlugin.java:1010)
	at com.hazelcast.map.impl.MapListenerAdaptors$4$1.onEvent(MapListenerAdaptors.java:128)
	at com.hazelcast.map.impl.MapListenerAdaptors$4$1.onEvent(MapListenerAdaptors.java:125)
	at com.hazelcast.map.impl.InternalMapListenerAdapter.onEvent(InternalMapListenerAdapter.java:56)
	at com.hazelcast.map.impl.InternalMapListenerAdapter.onEvent(InternalMapListenerAdapter.java:35)
	at com.hazelcast.map.impl.event.MapEventPublishingService.callListener(MapEventPublishingService.java:172)
	at com.hazelcast.map.impl.event.MapEventPublishingService.dispatchEntryEventData(MapEventPublishingService.java:184)
	at com.hazelcast.map.impl.event.MapEventPublishingService.dispatchEvent(MapEventPublishingService.java:83)
	at com.hazelcast.map.impl.event.MapEventPublishingService.dispatchEvent(MapEventPublishingService.java:47)
	at com.hazelcast.map.impl.MapService.dispatchEvent(MapService.java:106)
	at com.hazelcast.map.impl.MapService.dispatchEvent(MapService.java:77)
	at com.hazelcast.spi.impl.eventservice.impl.LocalEventDispatcher.run(LocalEventDispatcher.java:64)
	at com.hazelcast.util.executor.StripedExecutor$Worker.process(StripedExecutor.java:226)
	at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:209)

2020-04-14 09:43:00:239 INFO  [01901] Updated node status to 'ONLINE' [OHazelcastPlugin]
2020-04-14 09:43:00:239 INFO  [10.200.114.245]:2434 [orientdb] [3.10.6] [10.200.114.245]:2434 is SHUTTING_DOWN [LifecycleService]Uncaught exception in thread Thread-353
com.orientechnologies.orient.server.distributed.task.ODistributedOperationException: Request has been canceled
	at com.orientechnologies.orient.server.distributed.ODistributedResponseManagerImpl.waitForSynchronousResponses(ODistributedResponseManagerImpl.java:283)
	at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl.waitForResponse(ODistributedDatabaseImpl.java:1097)
	at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl.send2Nodes(ODistributedDatabaseImpl.java:549)
	at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl.send2Nodes(ODistributedDatabaseImpl.java:420)
	at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.sendRequest(ODistributedAbstractPlugin.java:630)
	at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.sendRequest(ODistributedAbstractPlugin.java:592)
	at com.orientechnologies.orient.server.distributed.impl.ODistributedLockManagerRequester.acquireExclusiveLock(ODistributedLockManagerRequester.java:78)
	at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.executeInDistributedDatabaseLock(ODistributedAbstractPlugin.java:1756)
	at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin$5.run(OHazelcastPlugin.java:1193)
	at java.lang.Thread.run(Thread.java:748)

2020-04-14 09:43:00:239 INFO  [01901]<-[01900] Received updated status 01900.ebrslxn-allevents2=ONLINE [OHazelcastPlugin]
2020-04-14 09:43:00:240 INFO  [01901] Distributed servers status (*=current @=lockmgr[01900]):

 [OHazelcastPlugin]
2020-04-14 09:43:00:246 SEVER [01901] Distributed database 'OSystem' not found [OHazelcastPlugin]
2020-04-14 09:43:00:246 WARNI [01901]->[01900] Lock Manager server '01900' went down during the request of locking resource 'ebrslxn-allevents2'. Waiting for the election of a new Lock Manager... [ODistributedLockManagerRequester]
2020-04-14 09:43:00:570 INFO  [10.200.114.245]:2434 [orientdb] [3.10.6] Shutting down multicast service... [Node]
2020-04-14 09:43:00:572 INFO  [10.200.114.245]:2434 [orientdb] [3.10.6] Shutting down connection manager... [Node]
2020-04-14 09:43:00:573 INFO  [10.200.114.245]:2434 [orientdb] [3.10.6] Connection[id=3, /10.200.114.245:47089->/10.200.114.244:2434, endpoint=[10.200.114.244]:2434, alive=false, type=MEMBER] closed. Reason: TcpIpConnectionManager is stopping [TcpIpConnection]
2020-04-14 09:43:00:574 INFO  [10.200.114.245]:2434 [orientdb] [3.10.6] Shutting down node engine... [Node]Hazelcast is not running
com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
	at com.hazelcast.spi.impl.operationservice.impl.InvocationRegistry.register(InvocationRegistry.java:128)
	at com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvoke(Invocation.java:554)
	at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke0(Invocation.java:525)
	at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke(Invocation.java:215)
	at com.hazelcast.spi.impl.operationservice.impl.InvocationBuilderImpl.invoke(InvocationBuilderImpl.java:60)
	at com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:424)
	at com.hazelcast.map.impl.proxy.MapProxySupport.getInternal(MapProxySupport.java:347)
	at com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:116)
	at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedMap.get(OHazelcastDistributedMap.java:60)
	at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage.getDistributedConfiguration(ODistributedStorage.java:1539)
	at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.getDatabaseConfiguration(ODistributedAbstractPlugin.java:481)
	at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.getDatabaseConfiguration(ODistributedAbstractPlugin.java:472)
	at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.executeInDistributedDatabaseLock(ODistributedAbstractPlugin.java:1761)
	at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.installDatabase(ODistributedAbstractPlugin.java:947)
	at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.entryUpdated(OHazelcastPlugin.java:1010)
	at com.hazelcast.map.impl.MapListenerAdaptors$4$1.onEvent(MapListenerAdaptors.java:128)
	at com.hazelcast.map.impl.MapListenerAdaptors$4$1.onEvent(MapListenerAdaptors.java:125)
	at com.hazelcast.map.impl.InternalMapListenerAdapter.onEvent(InternalMapListenerAdapter.java:56)
	at com.hazelcast.map.impl.InternalMapListenerAdapter.onEvent(InternalMapListenerAdapter.java:35)
	at com.hazelcast.map.impl.event.MapEventPublishingService.callListener(MapEventPublishingService.java:172)
	at com.hazelcast.map.impl.event.MapEventPublishingService.dispatchEntryEventData(MapEventPublishingService.java:184)
	at com.hazelcast.map.impl.event.MapEventPublishingService.dispatchEvent(MapEventPublishingService.java:83)
	at com.hazelcast.map.impl.event.MapEventPublishingService.dispatchEvent(MapEventPublishingService.java:47)
	at com.hazelcast.map.impl.MapService.dispatchEvent(MapService.java:106)
	at com.hazelcast.map.impl.MapService.dispatchEvent(MapService.java:77)
	at com.hazelcast.spi.impl.eventservice.impl.LocalEventDispatcher.run(LocalEventDispatcher.java:64)
	at com.hazelcast.util.executor.StripedExecutor$Worker.process(StripedExecutor.java:226)
	at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:209)
	at ------ submitted from ------.(Unknown Source)
	at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolve(InvocationFuture.java:127)
	at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveAndThrowIfException(InvocationFuture.java:79)
	at com.hazelcast.spi.impl.AbstractInvocationFuture.get(AbstractInvocationFuture.java:154)
	at com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:425)
	at com.hazelcast.map.impl.proxy.MapProxySupport.getInternal(MapProxySupport.java:347)
	at com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:116)
	at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedMap.get(OHazelcastDistributedMap.java:60)
	at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage.getDistributedConfiguration(ODistributedStorage.java:1539)
	at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.getDatabaseConfiguration(ODistributedAbstractPlugin.java:481)
	at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.getDatabaseConfiguration(ODistributedAbstractPlugin.java:472)
	at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.executeInDistributedDatabaseLock(ODistributedAbstractPlugin.java:1761)
	at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.installDatabase(ODistributedAbstractPlugin.java:947)
	at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.entryUpdated(OHazelcastPlugin.java:1010)
	at com.hazelcast.map.impl.MapListenerAdaptors$4$1.onEvent(MapListenerAdaptors.java:128)
	at com.hazelcast.map.impl.MapListenerAdaptors$4$1.onEvent(MapListenerAdaptors.java:125)
	at com.hazelcast.map.impl.InternalMapListenerAdapter.onEvent(InternalMapListenerAdapter.java:56)
	at com.hazelcast.map.impl.InternalMapListenerAdapter.onEvent(InternalMapListenerAdapter.java:35)
	at com.hazelcast.map.impl.event.MapEventPublishingService.callListener(MapEventPublishingService.java:172)
	at com.hazelcast.map.impl.event.MapEventPublishingService.dispatchEntryEventData(MapEventPublishingService.java:184)
	at com.hazelcast.map.impl.event.MapEventPublishingService.dispatchEvent(MapEventPublishingService.java:83)
	at com.hazelcast.map.impl.event.MapEventPublishingService.dispatchEvent(MapEventPublishingService.java:47)
	at com.hazelcast.map.impl.MapService.dispatchEvent(MapService.java:106)
	at com.hazelcast.map.impl.MapService.dispatchEvent(MapService.java:77)
	at com.hazelcast.spi.impl.eventservice.impl.LocalEventDispatcher.run(LocalEventDispatcher.java:64)
	at com.hazelcast.util.executor.StripedExecutor$Worker.process(StripedExecutor.java:226)
	at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:209)

2020-04-14 09:43:00:586 INFO  [10.200.114.245]:2434 [orientdb] [3.10.6] Destroying node NodeExtension. [NodeExtension]
2020-04-14 09:43:00:586 INFO  [10.200.114.245]:2434 [orientdb] [3.10.6] Hazelcast Shutdown is completed in 345 ms. [Node]
2020-04-14 09:43:00:587 INFO  [10.200.114.245]:2434 [orientdb] [3.10.6] [10.200.114.245]:2434 is SHUTDOWN [LifecycleService]
2020-04-14 09:43:00:588 INFO  [01901] Updated node status to 'OFFLINE' [OHazelcastPlugin]
2020-04-14 09:43:00:588 INFO  - teleporter [OServerPluginManager]
2020-04-14 09:43:00:588 INFO  - custom-sql-functions-manager [OServerPluginManager]
2020-04-14 09:43:00:588 INFO  - script-interpreter [OServerPluginManager]
2020-04-14 09:43:00:588 INFO  - automaticBackup [OServerPluginManager]
2020-04-14 09:43:00:588 INFO  - etl [OServerPluginManager]
2020-04-14 09:43:00:588 INFO  - neo4j-importer [OServerPluginManager]
2020-04-14 09:43:00:588 INFO  Shutting down databases: [OServer]
2020-04-14 09:43:00:588 INFO  Orient Engine is shutting down... [Orient]
2020-04-14 09:43:00:589 INFO  - shutdown storage: OSystem... [OrientDBDistributed]
2020-04-14 09:43:03:861 INFO  - shutdown storage: ebrslxn-allevents... [OrientDBDistributed]
2020-04-14 09:43:04:061 INFO  - shutdown storage: ebrslxn-allevents2... [OrientDBDistributed]
2020-04-14 09:43:04:243 INFO  - shutdown storage: esebtor-allevents... [OrientDBDistributed]
2020-04-14 09:43:04:425 INFO  - shutdown storage: allevents... [OrientDBDistributed]
2020-04-14 09:43:04:730 INFO  Clearing byte buffer pool [Orient]
2020-04-14 09:43:04:759 INFO  OrientDB Engine shutdown complete [Orient]
2020-04-14 09:43:04:759 INFO  OrientDB Server shutdown complete

This issue appears from time to time. I tried removing all the databases on the master to see if that helps, but that mostly cleared up the logs instead.

From 1902 (replica) we can see the following logs. What does it refer to by registered servers? I can see that all the nodes are still in the distributed-config.json, both for OSystem and the databases.

2020-04-27 12:09:56:531 INFO  [01902] Server '01900' was not found in the list of registered servers. Reloading configuration from cluster... [OClusterHealthChecker]
2020-04-27 12:09:56:531 WARNI [01902] Server '01900' was not found in the list of registered servers after the update, restarting the server... [OClusterHealthChecker]
2020-04-27 12:09:56:531 WARNI [01902] Sending request of restarting node '01900'... [OHazelcastPlugin]

Whereas the node 1900 (master) is logging the below.

2020-04-27 12:09:47:775 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] Established socket connection between /10.200.114.244:58013 and /10.200.114.245:2434 [TcpIpConnectionManager]
2020-04-27 12:09:47:796 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] We should merge to [10.200.114.245]:2434, because their data member count is bigger than ours [2 > 1] [ClusterJoinManager]
2020-04-27 12:09:47:796 WARNI [10.200.114.244]:2434 [orientdb] [3.10.6] [10.200.114.244]:2434 is merging [multicast] to [10.200.114.245]:2434 [MulticastJoiner]
2020-04-27 12:09:47:797 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] Locking cluster state. Initiator: [10.200.114.244]:2434, lease-time: 60000 [LockClusterStateOp]
2020-04-27 12:09:47:797 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] Extending cluster state lock. Initiator: [10.200.114.244]:2434, lease-time: 20000 [LockClusterStateOp]
2020-04-27 12:09:47:797 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] Changing cluster state from ClusterState{state=ACTIVE, lock=LockGuard{lockOwner=[10.200.114.244]:2434, lockOwnerId='da8ae9c7-91bd-44c9-b7f9-1cd33de4eb68', lockExpiryTime=1587982267797}} to ClusterStateChange{type=class com.hazelcast.cluster.ClusterState, newState=FROZEN}, initiator: [10.200.114.244]:2434, transient: false [CommitClusterStateOp]
2020-04-27 12:09:47:797 WARNI [10.200.114.244]:2434 [orientdb] [3.10.6] [10.200.114.244]:2434 is merging to [10.200.114.245]:2434, because: instructed by master [10.200.114.244]:2434 [MergeClustersOp]
2020-04-27 12:09:47:797 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] [10.200.114.244]:2434 is MERGING [LifecycleService]
2020-04-27 12:09:47:798 INFO  [01900] Updated node status to 'MERGING' [OHazelcastPlugin]
2020-04-27 12:09:47:798 WARNI [10.200.114.244]:2434 [orientdb] [3.10.6] Resetting local member UUID. Previous: da13c95c-b6ec-4d15-bd46-566a1e2be24b, new: 6429e357-ecd8-4324-8857-fbe3704c1bf7 [ClusterService]
2020-04-27 12:09:47:798 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] Connection[id=1, /10.200.114.244:58013->/10.200.114.245:2434, endpoint=[10.200.114.245]:2434, alive=false, type=MEMBER] closed. Reason: TcpIpConnectionManager is stopping [TcpIpConnection]
2020-04-27 12:09:47:808 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] Trying to join to discovered node: [10.200.114.245]:2434 [MulticastJoiner]
2020-04-27 12:09:47:809 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] Connecting to /10.200.114.245:2434, timeout: 0, bind-any: true [TcpIpConnector]
2020-04-27 12:09:47:809 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] Established socket connection between /10.200.114.244:35651 and /10.200.114.245:2434 [TcpIpConnectionManager]
2020-04-27 12:09:53:812 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] Cluster version set to 3.10 [system]
2020-04-27 12:09:53:813 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] Accepting socket connection from /10.200.114.246:50339 [TcpIpAcceptor]
2020-04-27 12:09:53:813 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] 

Members {size:3, ver:11} [
	Member [10.200.114.245]:2434 - 4efa46a1-5dc5-4abf-9e4d-0b8c960fe7e2
	Member [10.200.114.246]:2434 - 3d3ba90c-5ae0-42de-a85d-ae9b42379b2b
	Member [10.200.114.244]:2434 - 6429e357-ecd8-4324-8857-fbe3704c1bf7 this
]
 [ClusterService]
2020-04-27 12:09:53:813 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] Established socket connection between /10.200.114.244:2434 and /10.200.114.246:50339 [TcpIpConnectionManager]
2020-04-27 12:09:53:814 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] Connecting to /10.200.114.246:2434, timeout: 0, bind-any: true [TcpIpConnector]
2020-04-27 12:09:53:814 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] Established socket connection between /10.200.114.244:56593 and /10.200.114.246:2434 [TcpIpConnectionManager]
2020-04-27 12:09:53:815 INFO  [01900] Added new node id=Member [10.200.114.245]:2434 - 4efa46a1-5dc5-4abf-9e4d-0b8c960fe7e2 name=01901 [OHazelcastPlugin]
2020-04-27 12:09:53:821 INFO  [01900]<-[01901] Added node configuration id=Member [10.200.114.245]:2434 - 4efa46a1-5dc5-4abf-9e4d-0b8c960fe7e2 name=01901, now 2 nodes are configured [OHazelcastPlugin]
2020-04-27 12:09:53:823 INFO  [01900] Distributed servers status (*=current @=lockmgr[01900]):

+-----------+------+---------+-----+---------+-------------------+-------------------+------------------------+
|Name       |Status|Databases|Conns|StartedOn|Binary             |HTTP               |UsedMemory              |
+-----------+------+---------+-----+---------+-------------------+-------------------+------------------------+
|01901      |ONLINE|         |4    |09:37:35 |10.200.114.245:2424|10.200.114.245:2480|2.34GB/10.67GB (21.98%) |
|01900(*)(@)|ONLINE|         |1    |12:04:47 |10.200.114.244:2424|10.200.114.244:2480|589.53MB/10.67GB (5.40%)|
+-----------+------+---------+-----+---------+-------------------+-------------------+------------------------+
 [OHazelcastPlugin]
2020-04-27 12:09:53:823 INFO  [01900] Added new node id=Member [10.200.114.246]:2434 - 3d3ba90c-5ae0-42de-a85d-ae9b42379b2b name=01902 [OHazelcastPlugin]
2020-04-27 12:09:53:831 INFO  [01900]<-[01902] Added node configuration id=Member [10.200.114.246]:2434 - 3d3ba90c-5ae0-42de-a85d-ae9b42379b2b name=01902, now 3 nodes are configured [OHazelcastPlugin]
2020-04-27 12:09:53:833 INFO  [01900] Distributed servers status (*=current @=lockmgr[01900]):

+-----------+------+---------+-----+---------+-------------------+-------------------+------------------------+
|Name       |Status|Databases|Conns|StartedOn|Binary             |HTTP               |UsedMemory              |
+-----------+------+---------+-----+---------+-------------------+-------------------+------------------------+
|01902      |ONLINE|         |2    |09:45:57 |10.200.114.246:2424|10.200.114.246:2480|1.10GB/10.67GB (10.32%) |
|01901      |ONLINE|         |4    |09:37:35 |10.200.114.245:2424|10.200.114.245:2480|2.34GB/10.67GB (21.98%) |
|01900(*)(@)|ONLINE|         |1    |12:04:47 |10.200.114.244:2424|10.200.114.244:2480|589.53MB/10.67GB (5.40%)|
+-----------+------+---------+-----+---------+-------------------+-------------------+------------------------+
 [OHazelcastPlugin]
2020-04-27 12:09:53:847 INFO  [01900] Server '01902' was not found in the list of registered servers. Reloading configuration from cluster... [OClusterHealthChecker]
2020-04-27 12:09:54:815 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] [10.200.114.244]:2434 is MERGED [LifecycleService]
2020-04-27 12:09:54:816 INFO  [01900] Server merged the existent cluster, lockManager=01901, merging databases... [OHazelcastPlugin]
2020-04-27 12:09:54:816 INFO  [01900] Replacing old UUID da13c95c-b6ec-4d15-bd46-566a1e2be24b with the new 6429e357-ecd8-4324-8857-fbe3704c1bf7 [OHazelcastPlugin]
2020-04-27 12:09:54:820 INFO  [01900] Merging networks, waiting for the lockManager 01901 to be reachable... [OHazelcastPlugin$5]
2020-04-27 12:09:54:820 INFO  [01900] Merging networks, lockManager=01901 (active=true)... [OHazelcastPlugin$5]
2020-04-27 12:09:54:820 WARNI [01900] Network merged, lockManager=01901... [OHazelcastPlugin$5]
2020-04-27 12:09:54:820 INFO  [01900] Updated node status to 'ONLINE' [OHazelcastPlugin]
2020-04-27 12:09:54:821 INFO  [01900] Distributed servers status (*=current @=lockmgr[01901]):

+--------+-------+---------+-----+---------+-------------------+-------------------+------------------------+
|Name    |Status |Databases|Conns|StartedOn|Binary             |HTTP               |UsedMemory              |
+--------+-------+---------+-----+---------+-------------------+-------------------+------------------------+
|01902   |ONLINE |         |2    |09:45:57 |10.200.114.246:2424|10.200.114.246:2480|1.10GB/10.67GB (10.32%) |
|01901(@)|ONLINE |         |4    |09:37:35 |10.200.114.245:2424|10.200.114.245:2480|2.34GB/10.67GB (21.98%) |
|01900(*)|MERGING|         |0    |12:04:47 |10.200.114.244:2424|10.200.114.244:2480|632.38MB/10.67GB (5.79%)|
+--------+-------+---------+-----+---------+-------------------+-------------------+------------------------+
 [OHazelcastPlugin]
2020-04-27 12:09:56:533 WARNI [01900]<-[01902] Restarting server... [ORestartServerTask]
2020-04-27 12:09:56:533 INFO  [01900] Updated node status to 'OFFLINE' [OHazelcastPlugin]
2020-04-27 12:09:56:534 INFO  OrientDB Server is shutting down... [OServer]
2020-04-27 12:09:56:534 INFO  Shutting down listeners: [OServer]
2020-04-27 12:09:56:535 INFO  - ONetworkProtocolBinary /0.0.0.0:2424: [OServer]
2020-04-27 12:09:56:535 INFO  - ONetworkProtocolHttpDb /0.0.0.0:2480: [OServer]
2020-04-27 12:09:56:535 INFO  Shutting down protocols [OServer]
2020-04-27 12:09:56:537 INFO  Shutting down plugins: [OServerPluginManager]
2020-04-27 12:09:56:537 INFO  - studio [OServerPluginManager]
2020-04-27 12:09:56:537 INFO  - cluster [OServerPluginManager]
2020-04-27 12:09:56:537 WARNI Shutting down node '01900'... [OHazelcastPlugin]
2020-04-27 12:09:56:537 INFO  [01900] Updated node status to 'SHUTTINGDOWN' [OHazelcastPlugin]
2020-04-27 12:09:56:542 INFO  [01900] Received updated status 01900.ebrslxn-allevents=NOT_AVAILABLE [OHazelcastPlugin]
2020-04-27 12:09:56:544 INFO  [01900] Received updated status 01900.allevents=NOT_AVAILABLE [OHazelcastPlugin]
2020-04-27 12:09:56:545 INFO  [01900] Received updated status 01900.ebrslxn-allevents2=NOT_AVAILABLE [OHazelcastPlugin]
2020-04-27 12:09:56:552 INFO  [01900] Received updated status 01900.esebtor-allevents=NOT_AVAILABLE [OHazelcastPlugin]
2020-04-27 12:09:56:552 INFO  [01900] Distributed servers status (*=current @=lockmgr[01901]):

 [OHazelcastPlugin]
2020-04-27 12:09:56:552 INFO  [01900] Updated node status to 'OFFLINE' [OHazelcastPlugin]
2020-04-27 12:09:56:552 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] [10.200.114.244]:2434 is SHUTTING_DOWN [LifecycleService]
2020-04-27 12:09:56:901 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] Shutting down multicast service... [Node]
2020-04-27 12:09:56:903 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] Shutting down connection manager... [Node]
2020-04-27 12:09:56:904 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] Connection[id=3, /10.200.114.244:2434->/10.200.114.246:50339, endpoint=[10.200.114.246]:2434, alive=false, type=MEMBER] closed. Reason: TcpIpConnectionManager is stopping [TcpIpConnection]
2020-04-27 12:09:56:905 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] Connection[id=2, /10.200.114.244:35651->/10.200.114.245:2434, endpoint=[10.200.114.245]:2434, alive=false, type=MEMBER] closed. Reason: TcpIpConnectionManager is stopping [TcpIpConnection]
2020-04-27 12:09:56:905 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] Connection[id=4, /10.200.114.244:56593->/10.200.114.246:2434, endpoint=[10.200.114.246]:2434, alive=false, type=MEMBER] closed. Reason: TcpIpConnectionManager is stopping [TcpIpConnection]
2020-04-27 12:09:56:906 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] Shutting down node engine... [Node]
2020-04-27 12:09:56:914 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] Destroying node NodeExtension. [NodeExtension]
2020-04-27 12:09:56:914 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] Hazelcast Shutdown is completed in 362 ms. [Node]
2020-04-27 12:09:56:914 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] [10.200.114.244]:2434 is SHUTDOWN [LifecycleService]
2020-04-27 12:09:56:915 INFO  - teleporter [OServerPluginManager]
2020-04-27 12:09:56:915 INFO  - custom-sql-functions-manager [OServerPluginManager]
2020-04-27 12:09:56:915 INFO  - script-interpreter [OServerPluginManager]
2020-04-27 12:09:56:915 INFO  - etl [OServerPluginManager]
2020-04-27 12:09:56:915 INFO  - neo4j-importer [OServerPluginManager]
2020-04-27 12:09:56:915 INFO  Shutting down databases: [OServer]
2020-04-27 12:09:56:915 INFO  Orient Engine is shutting down... [Orient]
2020-04-27 12:09:56:915 INFO  - shutdown storage: OSystem... [OrientDBDistributed]
2020-04-27 12:09:57:126 INFO  Clearing byte buffer pool [Orient]
2020-04-27 12:09:57:150 INFO  OrientDB Engine shutdown complete [Orient]
2020-04-27 12:09:57:151 INFO  OrientDB Server shutdown complete
 [OServer]
2020-04-27 12:09:57:152 INFO  Detected limit of amount of simultaneously open files is 65536,  limit of open files for disk cache will be set to 32256 [ONative]
2020-04-27 12:09:57:152 WARNI Value of limit of simultaneously open files is too small, recommended value is 262144 [ONative]
2020-04-27 12:09:57:152 INFO  OrientDB Server v3.0.21 - Veloce (build 1d8b0c01ca8fe12a516bba355681e3ec8b218d1d, branch 3.0.x) is starting up... [OServer]
2020-04-27 12:09:57:153 INFO  System is started under an effective user : `flocxdev` [OEngineLocalPaginated]
2020-04-27 12:09:57:153 INFO  Allocation of 62259 pages. [OEngineLocalPaginated]
2020-04-27 12:09:57:384 INFO  WAL maximum segment size is set to 6,885 MB [OrientDBDistributed]
2020-04-27 12:09:57:384 INFO  Databases directory: /proj/flocx_dev/seroius01900/orientdb [OServer]
2020-04-27 12:09:57:390 INFO  Page size for WAL located in /local/apps/orientdb/orientdb-soft-link/wal is set to 4096 bytes. [OCASDiskWriteAheadLog]
2020-04-27 12:09:57:505 INFO  Storage 'plocal:/proj/flocx_dev/seroius01900/orientdb/OSystem' is opened under OrientDB distribution : 3.0.21 - Veloce (build 1d8b0c01ca8fe12a516bba355681e3ec8b218d1d, branch 3.0.x) [OLocalPaginatedStorage]
2020-04-27 12:09:57:520 INFO  Listening binary connections on 0.0.0.0:2424 (protocol v.37, socket=default) [OServerNetworkListener]
2020-04-27 12:09:57:521 INFO  Listening http connections on 0.0.0.0:2480 (protocol v.10, socket=default) [OServerNetworkListener]
2020-04-27 12:09:57:522 INFO  Installing dynamic plugin 'orientdb-teleporter-3.0.21.jar'... [OServerPluginManager]
2020-04-27 12:09:57:523 INFO  Installing dynamic plugin 'orientdb-neo4j-importer-plugin-3.0.21-dist.jar'... [OServerPluginManager]
2020-04-27 12:09:57:524 INFO  Installing dynamic plugin 'orientdb-etl-3.0.21.jar'... [OServerPluginManager]
2020-04-27 12:09:57:524 INFO  Installing dynamic plugin 'orientdb-studio-3.0.21.zip'... [OServerPluginManager]
2020-04-27 12:09:57:525 INFO  ODefaultPasswordAuthenticator is active [ODefaultPasswordAuthenticator]
2020-04-27 12:09:57:525 INFO  OServerConfigAuthenticator is active [OServerConfigAuthenticator]
2020-04-27 12:09:57:525 INFO  OSystemUserAuthenticator is active [OSystemUserAuthenticator]
2020-04-27 12:09:57:527 INFO  Configuring Hazelcast from '/local/apps/orientdb/config/hazelcast.xml'. [FileSystemXmlConfig]
2020-04-27 12:09:57:532 WARNI Name of the hazelcast schema location is incorrect, using default [AbstractXmlConfigHelper]
2020-04-27 12:09:57:560 INFO  [LOCAL] [orientdb] [3.10.6] Prefer IPv4 stack is true. [AddressPicker]
2020-04-27 12:09:57:560 INFO  [LOCAL] [orientdb] [3.10.6] Picked [10.200.114.244]:2434, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=2434], bind any local is true [AddressPicker]
2020-04-27 12:09:57:561 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] Hazelcast 3.10.6 (20181010 - fdcf6d6) starting at [10.200.114.244]:2434 [system]
2020-04-27 12:09:57:561 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] Copyright (c) 2008-2018, Hazelcast, Inc. All Rights Reserved. [system]
2020-04-27 12:09:57:561 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] Configured Hazelcast Serialization version: 1 [system]
2020-04-27 12:09:57:561 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] A non-empty group password is configured for the Hazelcast member. Starting with Hazelcast version 3.8.2, members with the same group name, but with different group passwords (that do not use authentication) form a cluster. The group password configuration will be removed completely in a future release. [Node]
2020-04-27 12:09:57:564 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] Backpressure is disabled [BackpressureRegulator]
2020-04-27 12:09:57:564 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] Running with 2 response threads [InboundResponseHandlerSupplier]
2020-04-27 12:09:57:572 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] Creating MulticastJoiner [Node]
2020-04-27 12:09:57:577 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] Starting 1 partition threads and 2 generic threads (1 dedicated for priority tasks) [OperationExecutorImpl]
2020-04-27 12:09:57:577 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments. [Diagnostics]
2020-04-27 12:09:57:577 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] [10.200.114.244]:2434 is STARTING [LifecycleService]
2020-04-27 12:09:59:861 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] Cluster version set to 3.10 [system]
2020-04-27 12:09:59:861 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] 

Members {size:1, ver:1} [
	Member [10.200.114.244]:2434 - d8241552-c353-4a7a-857d-6691f161ee0f this
]
 [ClusterService]
2020-04-27 12:09:59:862 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] [10.200.114.244]:2434 is STARTED [LifecycleService]
2020-04-27 12:09:59:862 INFO  Starting distributed server '01900' (hzID=d8241552-c353-4a7a-857d-6691f161ee0f)... [OHazelcastPlugin]
2020-04-27 12:09:59:862 INFO  Distributed cluster time=Mon Apr 27 12:09:59 CEST 2020 (delta from local node=0)... [OHazelcastPlugin]
2020-04-27 12:09:59:863 INFO  [10.200.114.244]:2434 [orientdb] [3.10.6] Initializing cluster partition table arrangement... [PartitionStateManager]
2020-04-27 12:09:59:870 INFO  [01900] Registered local server with nodeId=0 [OHazelcastPlugin]
2020-04-27 12:09:59:873 INFO  Distributed Lock Manager server is '01900' [OHazelcastPlugin]
2020-04-27 12:09:59:873 INFO  [01900] Loaded configuration for database 'OSystem' from disk: /local/apps/orientdb/config/default-distributed-db-config.json [ODistributedStorage]
2020-04-27 12:09:59:881 INFO  [01900] Servers in cluster: [01900] [OHazelcastPlugin]
2020-04-27 12:09:59:886 INFO  [01900] Updated node status to 'ONLINE' [OHazelcastPlugin]
2020-04-27 12:09:59:888 INFO  [01900] Distributed servers status (*=current @=lockmgr[01900]):

+-----------+------+---------+-----+---------+-------------------+-------------------+------------------------+
|Name       |Status|Databases|Conns|StartedOn|Binary             |HTTP               |UsedMemory              |
+-----------+------+---------+-----+---------+-------------------+-------------------+------------------------+
|01900(*)(@)|ONLINE|         |0    |12:09:57 |10.200.114.244:2424|10.200.114.244:2480|514.92MB/10.67GB (4.71%)|
+-----------+------+---------+-----+---------+-------------------+-------------------+------------------------+
 [OHazelcastPlugin]
2020-04-27 12:09:59:888 WARNI Authenticated clients can execute any kind of code into the server by using the following allowed languages: [sql] [OServerSideScriptInterpreter]
2020-04-27 12:09:59:888 INFO  OrientDB Studio available at http://10.200.114.244:2480/studio/index.html [OServer]
2020-04-27 12:09:59:888 INFO  OrientDB Server is active v3.0.21 - Veloce (build 1d8b0c01ca8fe12a516bba355681e3ec8b218d1d, branch 3.0.x). [OServer]

Any help on this would be very appreciated!