Why is orientdb always requesting FULL sync in distributed mode?

If a node is discovered in distributed mode it is always requesting a FULL sync of the database, no matter if it already contains the data or not. Even if I copy the exact same database from one one to the other on filesystem, the full sync happens. Is this expected behavour ? I would expect a delta sync only if some pieces are missing. See the logs of a joining node below:

2020-04-28 07:41:23:719 INFO  [tni-example20-odb-orientdb-1] Current node is a MASTER for database 'example' [ODistributedAbstractPlugin$3]
2020-04-28 07:41:23:732 WARNI [tni-example20-odb-orientdb-1]->[[tni-example20-odb-orientdb-0]] requesting delta database sync for 'example' on local server... [OHazelcastPlugin]
2020-04-28 07:41:23:741 INFO  Direct IO for WAL located in /orientdb/backup is allowed with block size 4096 bytes. [OCASDiskWriteAheadLog]
2020-04-28 07:41:23:742 INFO  Page size for WAL located in /orientdb/backup is set to 4096 bytes. [OCASDiskWriteAheadLog]
2020-04-28 07:41:24:117 INFO  Storage 'plocal:/orientdb/databases/example' is opened under OrientDB distribution : 3.0.27 - Veloce (build e4e73e77074c4f49337ae58b171513cd10ac1157, branch UNKNOWN) [OLocalPaginatedStorage]
2020-04-28 07:41:24:294 WARNI [tni-example20-odb-orientdb-1] Node 'tni-example20-odb-orientdb-1' is NOT_AVAILABLE, the command 'sql.select from OUser where name = ? limit 1' against database 'example' will be executed only on local server with the possibility to have partial result [ODistributedStorage]
2020-04-28 07:41:24:308 INFO  Found 2 records changed in last 2 operations [OLocalPaginatedStorage]
2020-04-28 07:41:24:309 INFO  [tni-example20-odb-orientdb-1] Executing the realignment of the last records modified before last close [#0:11, #0:370]... [OHazelcastPlugin]
2020-04-28 07:41:24:313 WARNI [tni-example20-odb-orientdb-1] Node 'tni-example20-odb-orientdb-1' is NOT_AVAILABLE, the command 'sql.select from OUser where name = ? limit 1' against database 'example' will be executed only on local server with the possibility to have partial result [ODistributedStorage]
2020-04-28 07:41:24:318 INFO  [tni-example20-odb-orientdb-1] Realignment completed. [OHazelcastPlugin]
2020-04-28 07:41:24:321 INFO  [tni-example20-odb-orientdb-1]->[tni-example20-odb-orientdb-0] Requesting database delta sync for 'example' LSN=OLogSequenceNumber{segment=2033, position=13005797}... [OHazelcastPlugin]
2020-04-28 07:41:24:350 WARNI [tni-example20-odb-orientdb-1]<-[tni-example20-odb-orientdb-0] Error on installing database delta for 'example' (err=Requested delta sync with LSN=OLogSequenceNumber{segment=2033, position=13005797} but found the following error: Requested database delta sync with LSN=OLogSequenceNumber{segment=2033, position=13005797} but not found in database) [OHazelcastPlugin]
2020-04-28 07:41:24:350 INFO  [tni-example20-odb-orientdb-1] Requesting full sync for database 'example'... [OHazelcastPlugin]
2020-04-28 07:41:24:354 INFO  [tni-example20-odb-orientdb-1]->[[tni-example20-odb-orientdb-0]] Requesting deploy of database 'example' on local server... [OHazelcastPlugin]
2020-04-28 07:41:29:945 WARNI [tni-example20-odb-orientdb-1] Moving existent database 'example' in '/orientdb/databases/example' to '/orientdb/backup/databases/example' and get a fresh copy from a remote node... [OHazelcastPlugin]
2020-04-28 07:41:29:948 WARNI [tni-example20-odb-orientdb-1] Error on moving existent database 'example' located in '/orientdb/databases/example' to '/orientdb/backup/databases/example' (error=java.nio.file.DirectoryNotEmptyException: /orientdb/databases/example). Deleting old database anyway [OHazelcastPlugin]
2020-04-28 07:41:30:484 INFO  [tni-example20-odb-orientdb-1]<-[tni-example20-odb-orientdb-0] Installing database 'example' to: /orientdb/databases/example... [OHazelcastPlugin]
2020-04-28 07:41:30:486 INFO  [tni-example20-odb-orientdb-1] - writing chunk #1 offset=0 size=8.00MB [OSyncReceiver]
2020-04-28 07:41:33:318 INFO  [tni-example20-odb-orientdb-1] - writing chunk #2 offset=0 size=8.00MB [OSyncReceiver]
2020-04-28 07:41:37:047 INFO  [tni-example20-odb-orientdb-1] - writing chunk #3 offset=0 size=8.00MB [OSyncReceiver]
2020-04-28 07:41:39:296 INFO  [172.17.43.183]:2434 [hz-orientdb-tni] [3.11] Initialized new cluster connection between /172.17.43.183:2434 and /172.17.89.77:60602 [TcpIpConnection]
2020-04-28 07:41:39:841 INFO  [tni-example20-odb-orientdb-1] - writing chunk #4 offset=0 size=8.00MB [OSyncReceiver]
2020-04-28 07:41:42:554 INFO  [tni-example20-odb-orientdb-1] - writing chunk #5 offset=0 size=8.00MB [OSyncReceiver]
2020-04-28 07:41:44:274 INFO  [tni-example20-odb-orientdb-1] - writing chunk #6 offset=0 size=8.00MB [OSyncReceiver]
2020-04-28 07:41:46:291 INFO  [172.17.43.183]:2434 [hz-orientdb-tni] [3.11] 

Members {size:3, ver:3} [
	Member [172.17.17.65]:2434 - c0e7f371-e488-4d36-8ac6-43719c37cbf0
	Member [172.17.43.183]:2434 - 485f48d7-beba-486a-8225-289d095f6cc7 this
	Member [172.17.89.77]:2434 - f427f582-ddd2-454c-93ee-29b5a9cdc090
]
 [ClusterService]
2020-04-28 07:41:47:018 INFO  [tni-example20-odb-orientdb-1] - writing chunk #7 offset=0 size=8.00MB [OSyncReceiver]
2020-04-28 07:41:48:805 INFO  [tni-example20-odb-orientdb-1] - writing chunk #8 offset=0 size=8.00MB [OSyncReceiver]
2020-04-28 07:41:51:509 INFO  [tni-example20-odb-orientdb-1] - writing chunk #9 offset=0 size=8.00MB [OSyncReceiver]

when you restart a node with config “autoDeploy”: true, it will auto re deploy

1 Like

Yes but the point is, why is it always syncing the whole db?

I’ve wondered the same thing myself. For every node we add or even restart, it does a full sync every time. I’m assuming that’s because the nodes have not recorded something to understand their state with respect to the replication. Hoping that improves in 3.1 because a large DB sync every time is painful, and not really necessary.

your delta data in that note too large , so it require resysnc whole db.or it not found a check point

I have similar question.
My distributed cluster (v3.0.32) runs inside a kubernetes cluster, and there are 3 nodes, let’s call them node-0, node-1, and node-2. When I patched the statefulset, the first node been taken down is node-2, which did a successful delta sync from node-0, total 173 chunks. Then node-1 was taken down, at startup, it synced 273 chunks (delta) from node-2. The last one was node-0 which was trying to do a delta sync, after 343 chunks been written, it errors out and requested full sync (see logs below).
I do a graceful shutdown when terminating pods, and each have proper liveness/readiness probes to ensure not taking another out until the user database can be connected.

2020-06-29 18:02:08:176 INFO [node-0]->[node-2] Requesting database delta sync for ‘my-user-database’ LSN=OLogSequenceNumber{segment=547, position=10260115}… [OHazelcastPlugin]
2020-06-29 18:02:08:586 INFO [node-0] Node is not online yet (status=STARTING), blocking the command until it is online (retry=33, queue=5 worker=0) [ODistributedWorker]
2020-06-29 18:02:08:587 INFO [node-0] Node is not online yet (status=STARTING), blocking the command until it is online (retry=43, queue=5 worker=-4) [ODistributedWorker]
2020-06-29 18:02:09:060 INFO [node-0]<-[node-2] Installing database ‘my-user-database’ to: /orientdb/user_db/my-user-database… [OHazelcastPlugin]
2020-06-29 18:02:09:061 INFO [node-0] - writing chunk #1 offset=0 size=1024b [OSyncReceiver]
2020-06-29 18:02:09:063 INFO [node-0] - writing chunk #2 offset=0 size=1024b [OSyncReceiver]
2020-06-29 18:02:09:066 INFO [node-0]<-[node-2] Started import of delta for database ‘my-user-database’ [OIncrementalServerSync$1]
2020-06-29 18:02:09:082 INFO [node-0] - writing chunk #3 offset=0 size=1024b [OSyncReceiver]
2020-06-29 18:02:09:108 INFO [node-0] - writing chunk #4 offset=0 size=1024b [OSyncReceiver]
2020-06-29 18:02:09:113 INFO [node-0] - writing chunk #5 offset=0 size=1024b [OSyncReceiver]


2020-06-29 18:02:10:117 INFO [node-0] - writing chunk #340 offset=0 size=1024b [OSyncReceiver]
2020-06-29 18:02:10:125 INFO [node-0] - writing chunk #341 offset=0 size=1024b [OSyncReceiver]
2020-06-29 18:02:10:134 INFO [node-0] - writing chunk #342 offset=0 size=1024b [OSyncReceiver]
2020-06-29 18:02:10:142 INFO [node-0] - writing chunk #343 offset=0 size=1024b [OSyncReceiver]


2020-06-29 18:02:10:586 INFO [node-0] Node is not online yet (status=STARTING), blocking the command until it is online (retry=34, queue=5 worker=0) [ODistributedWorker]
2020-06-29 18:02:10:587 INFO [node-0] Node is not online yet (status=STARTING), blocking the command until it is online (retry=44, queue=5 worker=-4) [ODistributedWorker]
2020-06-29 18:02:11:149 INFO [node-0]<-[node-2] - 515 total entries: 0 created, 515 updated, 0 deleted, 0 holes, 0 skipped… [OIncrementalServerSync$1]
2020-06-29 18:02:12:586 INFO [node-0] Node is not online yet (status=STARTING), blocking the command until it is online (retry=35, queue=5 worker=0) [ODistributedWorker]
2020-06-29 18:02:12:588 INFO [node-0] Node is not online yet (status=STARTING), blocking the command until it is online (retry=45, queue=5 worker=-4) [ODistributedWorker]
2020-06-29 18:02:14:587 INFO [node-0] Node is not online yet (status=STARTING), blocking the command until it is online (retry=36, queue=5 worker=0) [ODistributedWorker]


2020-06-29 18:03:08:600 INFO [node-0] Node is not online yet (status=STARTING), blocking the command until it is online (retry=73, queue=6 worker=-4) [ODistributedWorker][node-0]<-[node-2] Error on installing database delta ‘my-user-database’ on local server
java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:197)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at com.orientechnologies.orient.server.distributed.impl.OIncrementalServerSync$1.call(OIncrementalServerSync.java:142)
at com.orientechnologies.orient.core.db.OScenarioThreadLocal.executeAsDistributed(OScenarioThreadLocal.java:69)
at com.orientechnologies.orient.server.distributed.impl.OIncrementalServerSync.importDelta(OIncrementalServerSync.java:78)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin$6.call(ODistributedAbstractPlugin.java:1988)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin$6.call(ODistributedAbstractPlugin.java:1936)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.executeInDistributedDatabaseLock(ODistributedAbstractPlugin.java:1776)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.installDatabaseOnLocalNode(ODistributedAbstractPlugin.java:1936)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.installDatabaseFromNetwork(ODistributedAbstractPlugin.java:1603)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.installResponseDeltaSync(ODistributedAbstractPlugin.java:1264)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.requestDatabaseDelta(ODistributedAbstractPlugin.java:1187)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin$3.call(ODistributedAbstractPlugin.java:1004)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin$3.call(ODistributedAbstractPlugin.java:948)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.executeInDistributedDatabaseLock(ODistributedAbstractPlugin.java:1776)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.installDatabase(ODistributedAbstractPlugin.java:947)
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installNewDatabasesFromCluster(OHazelcastPlugin.java:1435)
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.startup(OHazelcastPlugin.java:291)
at com.orientechnologies.orient.server.OServer.registerPlugins(OServer.java:1194)
at com.orientechnologies.orient.server.OServer.activate(OServer.java:469)
at com.orientechnologies.orient.server.OServerMain$1.run(OServerMain.java:48)
[node-0]<-[node-2] Error while applying changes of database delta sync on ‘my-user-database’: forcing full database sync…
com.orientechnologies.orient.server.distributed.ODistributedException: Error on installing database delta ‘my-user-database’ on local server
at com.orientechnologies.orient.server.distributed.impl.OIncrementalServerSync$1.call(OIncrementalServerSync.java:262)
at com.orientechnologies.orient.core.db.OScenarioThreadLocal.executeAsDistributed(OScenarioThreadLocal.java:69)
at com.orientechnologies.orient.server.distributed.impl.OIncrementalServerSync.importDelta(OIncrementalServerSync.java:78)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin$6.call(ODistributedAbstractPlugin.java:1988)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin$6.call(ODistributedAbstractPlugin.java:1936)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.executeInDistributedDatabaseLock(ODistributedAbstractPlugin.java:1776)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.installDatabaseOnLocalNode(ODistributedAbstractPlugin.java:1936)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.installDatabaseFromNetwork(ODistributedAbstractPlugin.java:1603)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.installResponseDeltaSync(ODistributedAbstractPlugin.java:1264)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.requestDatabaseDelta(ODistributedAbstractPlugin.java:1187)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin$3.call(ODistributedAbstractPlugin.java:1004)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin$3.call(ODistributedAbstractPlugin.java:948)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.executeInDistributedDatabaseLock(ODistributedAbstractPlugin.java:1776)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.installDatabase(ODistributedAbstractPlugin.java:947)
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installNewDatabasesFromCluster(OHazelcastPlugin.java:1435)
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.startup(OHazelcastPlugin.java:291)
at com.orientechnologies.orient.server.OServer.registerPlugins(OServer.java:1194)
at com.orientechnologies.orient.server.OServer.activate(OServer.java:469)
at com.orientechnologies.orient.server.OServerMain$1.run(OServerMain.java:48)
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:197)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at com.orientechnologies.orient.server.distributed.impl.OIncrementalServerSync$1.call(OIncrementalServerSync.java:142)
… 18 more
Error installing database from network
com.orientechnologies.orient.server.distributed.task.ODistributedDatabaseDeltaSyncException: Error while applying changes of database delta sync on ‘my-user-database’: forcing full database sync…
at com.orientechnologies.orient.server.distributed.impl.OIncrementalServerSync.importDelta(OIncrementalServerSync.java:282)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin$6.call(ODistributedAbstractPlugin.java:1988)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin$6.call(ODistributedAbstractPlugin.java:1936)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.executeInDistributedDatabaseLock(ODistributedAbstractPlugin.java:1776)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.installDatabaseOnLocalNode(ODistributedAbstractPlugin.java:1936)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.installDatabaseFromNetwork(ODistributedAbstractPlugin.java:1603)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.installResponseDeltaSync(ODistributedAbstractPlugin.java:1264)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.requestDatabaseDelta(ODistributedAbstractPlugin.java:1187)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin$3.call(ODistributedAbstractPlugin.java:1004)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin$3.call(ODistributedAbstractPlugin.java:948)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.executeInDistributedDatabaseLock(ODistributedAbstractPlugin.java:1776)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.installDatabase(ODistributedAbstractPlugin.java:947)
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installNewDatabasesFromCluster(OHazelcastPlugin.java:1435)
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.startup(OHazelcastPlugin.java:291)
at com.orientechnologies.orient.server.OServer.registerPlugins(OServer.java:1194)
at com.orientechnologies.orient.server.OServer.activate(OServer.java:469)
at com.orientechnologies.orient.server.OServerMain$1.run(OServerMain.java:48)
Caused by: com.orientechnologies.orient.server.distributed.ODistributedException: Error on installing database delta ‘my-user-database’ on local server
at com.orientechnologies.orient.server.distributed.impl.OIncrementalServerSync$1.call(OIncrementalServerSync.java:262)
at com.orientechnologies.orient.core.db.OScenarioThreadLocal.executeAsDistributed(OScenarioThreadLocal.java:69)
at com.orientechnologies.orient.server.distributed.impl.OIncrementalServerSync.importDelta(OIncrementalServerSync.java:78)
… 16 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:197)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at com.orientechnologies.orient.server.distributed.impl.OIncrementalServerSync$1.call(OIncrementalServerSync.java:142)
… 18 more

2020-06-29 18:03:10:157 INFO [node-0]->[node-1] Requesting database delta sync for ‘my-user-database’ LSN=OLogSequenceNumber{segment=550, position=34553480}… [OHazelcastPlugin]
2020-06-29 18:03:10:175 WARNI [node-0]<-[node-1] Error on installing database delta for ‘my-user-database’ (err=Requested delta sync with LSN=OLogSequenceNumber{segment=550, position=34553480} but found the following error: Requested database delta sync with LSN=OLogSequenceNumber{segment=550, position=34553480} but not found in database) [OHazelcastPlugin]
2020-06-29 18:03:10:175 INFO [node-0] Requesting full sync for database ‘my-user-database’… [OHazelcastPlugin]
2020-06-29 18:03:10:175 INFO [node-0]->[[node-1]] Requesting deploy of database ‘my-user-database’ on local server… [OHazelcastPlugin]