distributed sync of big databases is requested multiple times

[madmac2501]

OrientDB Version: 3.1.2-SNAPSHOT including commit 7ab7ca2

Java Version: 1.8

OS: Linux

Expected behavior

database synchronization finishes

Actual behavior

database synchronization is requested multiple times in parallel and doesn't finish

Steps to reproduce

setup 3 distributed nodes in master master
insert data until the database is 8.5GB in disk
stop one instance and start again
synchronization process is started automatically

2020-07-17 11:25:35:866 INFO [oriendb-cluster-1-demo-cloud-se-aws-us-east-1]->[[oriendb-cluster-2-demo-cloud-se-aws-us-east-1]] Requesting deploy of database 'xxx' on local server... [OHazelcastPlugin]
2020-07-17 11:25:35:946 INFO [oriendb-cluster-1-demo-cloud-se-aws-us-east-1] - writing chunk #11 offset=0 size=8.00MB [OSyncReceiver]
2020-07-17 11:25:36:960 INFO [oriendb-cluster-1-demo-cloud-se-aws-us-east-1] - writing chunk #20 offset=0 size=8.00MB [OSyncReceiver]
2020-07-17 11:25:38:974 INFO [oriendb-cluster-1-demo-cloud-se-aws-us-east-1] - writing chunk #5 offset=0 size=8.00MB [OSyncReceiver]
2020-07-17 11:25:39:987 INFO [oriendb-cluster-1-demo-cloud-se-aws-us-east-1] - writing chunk #8 offset=0 size=8.00MB [OSyncReceiver]
2020-07-17 11:25:42:001 INFO [oriendb-cluster-1-demo-cloud-se-aws-us-east-1] - writing chunk #3 offset=0 size=8.00MB [OSyncReceiver]
2020-07-17 11:25:44:029 INFO [oriendb-cluster-1-demo-cloud-se-aws-us-east-1]<-[oriendb-cluster-2-demo-cloud-se-aws-us-east-1] Installing database 'xxx' to: /opt/orientdb/databases/xxx... [OHazelcastPlugin]
2020-07-17 11:25:44:029 INFO [oriendb-cluster-1-demo-cloud-se-aws-us-east-1] - writing chunk #1 offset=0 size=8.00MB [OSyncReceiver]
2020-07-17 11:25:44:030 SEVER Cannot delete storage directory with path /opt/orientdb/databases/xxx because directory is not empty. Files: [/opt/orientdb/databases/xxx/distributed-config.json, /opt/orientdb/databases/xxx/distributed-sync.json, /opt/orientdb/databases/xxx/command-cache.json] [OLocalPaginatedStorage]Error installing database from network
com.orientechnologies.orient.core.exception.ODatabaseException: Cannot create database 'xxx'
at com.orientechnologies.orient.core.db.OrientDBEmbedded.networkRestore(OrientDBEmbedded.java:668)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin$7.call(ODistributedAbstractPlugin.java:2216)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin$7.call(ODistributedAbstractPlugin.java:2136)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.executeInDistributedDatabaseLock(ODistributedAbstractPlugin.java:1916)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.installDatabaseOnLocalNode(ODistributedAbstractPlugin.java:2132)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.installDatabaseFromNetwork(ODistributedAbstractPlugin.java:1713)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.requestDatabaseFullSync(ODistributedAbstractPlugin.java:1493)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.requestFullDatabase(ODistributedAbstractPlugin.java:1269)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin$4.call(ODistributedAbstractPlugin.java:1100)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin$4.call(ODistributedAbstractPlugin.java:1006)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.executeInDistributedDatabaseLock(ODistributedAbstractPlugin.java:1916)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.installDatabase(ODistributedAbstractPlugin.java:1002)
at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl.validateStatus(ODistributedDatabaseImpl.java:1339)
at com.orientechnologies.orient.server.distributed.impl.task.OUpdateDatabaseSequenceStatusTask.execute(OUpdateDatabaseSequenceStatusTask.java:41)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin$2.call(ODistributedAbstractPlugin.java:686)
at com.orientechnologies.orient.core.db.OScenarioThreadLocal.executeAsDistributed(OScenarioThreadLocal.java:91)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.executeOnLocalNode(ODistributedAbstractPlugin.java:681)
at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl.execute(ODistributedDatabaseImpl.java:424)
at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl.lambda$processRequest$3(ODistributedDatabaseImpl.java:395)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.CancellationException
at java.util.concurrent.FutureTask.report(FutureTask.java:121)
at java.util.concurrent.FutureTask.get(FutureTask.java:206)
at com.orientechnologies.orient.core.db.viewmanager.ViewManager.close(ViewManager.java:177)
at com.orientechnologies.orient.server.distributed.impl.metadata.OSharedContextDistributed.close(OSharedContextDistributed.java:91)
at com.orientechnologies.orient.core.db.OrientDBEmbedded.networkRestore(OrientDBEmbedded.java:655)
... 23 more

2020-07-17 11:25:46:041 INFO [oriendb-cluster-1-demo-cloud-se-aws-us-east-1] - writing chunk #12 offset=0 size=8.00MB [OSyncReceiver]
2020-07-17 11:25:47:055 INFO [oriendb-cluster-1-demo-cloud-se-aws-us-east-1] - writing chunk #21 offset=0 size=8.00MB [OSyncReceiver]
2020-07-17 11:25:49:069 INFO [oriendb-cluster-1-demo-cloud-se-aws-us-east-1] - writing chunk #6 offset=0 size=8.00MB [OSyncReceiver]
2020-07-17 11:25:50:082 INFO [oriendb-cluster-1-demo-cloud-se-aws-us-east-1] - writing chunk #9 offset=0 size=8.00MB [OSyncReceiver]
2020-07-17 11:25:52:096 INFO [oriendb-cluster-1-demo-cloud-se-aws-us-east-1] - writing chunk #4 offset=0 size=8.00MB [OSyncReceiver]
2020-07-17 11:25:54:030 INFO [oriendb-cluster-1-demo-cloud-se-aws-us-east-1]->[[oriendb-cluster-2-demo-cloud-se-aws-us-east-1]] Requesting deploy of database 'xxx' on local server... [OHazelcastPlugin]
2020-07-17 11:25:54:109 INFO [oriendb-cluster-1-demo-cloud-se-aws-us-east-1] - writing chunk #2 offset=0 size=8.00MB [OSyncReceiver]
2020-07-17 11:25:55:123 INFO [oriendb-cluster-1-demo-cloud-se-aws-us-east-1] - writing chunk #13 offset=0 size=8.00MB [OSyncReceiver]
2020-07-17 11:25:57:136 INFO [oriendb-cluster-1-demo-cloud-se-aws-us-east-1] - writing chunk #22 offset=0 size=8.00MB [OSyncReceiver]
2020-07-17 11:25:58:150 INFO [oriendb-cluster-1-demo-cloud-se-aws-us-east-1] - writing chunk #7 offset=0 size=8.00MB [OSyncReceiver]
2020-07-17 11:26:00:163 INFO [oriendb-cluster-1-demo-cloud-se-aws-us-east-1] - writing chunk #10 offset=0 size=8.00MB [OSyncReceiver]
2020-07-17 11:26:02:177 INFO [oriendb-cluster-1-demo-cloud-se-aws-us-east-1] - writing chunk #5 offset=0 size=8.00MB [OSyncReceiver]
2020-07-17 11:26:04:205 INFO [oriendb-cluster-1-demo-cloud-se-aws-us-east-1]<-[oriendb-cluster-2-demo-cloud-se-aws-us-east-1] Installing database 'xxx' to: /opt/orientdb/databases/xxx... [OHazelcastPlugin]
2020-07-17 11:26:04:206 INFO [oriendb-cluster-1-demo-cloud-se-aws-us-east-1] - writing chunk #1 offset=0 size=8.00MB [OSyncReceiver]
2020-07-17 11:26:04:206 SEVER Cannot delete storage directory with path /opt/orientdb/databases/xxx because directory is not empty. Files: [/opt/orientdb/databases/xxx/distributed-config.json, /opt/orientdb/databases/xxx/distributed-sync.json, /opt/orientdb/databases/xxx/command-cache.json] [OLocalPaginatedStorage]Error installing database from network
com.orientechnologies.orient.core.exception.ODatabaseException: Cannot create database 'xxx'
at com.orientechnologies.orient.core.db.OrientDBEmbedded.networkRestore(OrientDBEmbedded.java:668)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin$7.call(ODistributedAbstractPlugin.java:2216)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin$7.call(ODistributedAbstractPlugin.java:2136)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.executeInDistributedDatabaseLock(ODistributedAbstractPlugin.java:1916)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.installDatabaseOnLocalNode(ODistributedAbstractPlugin.java:2132)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.installDatabaseFromNetwork(ODistributedAbstractPlugin.java:1713)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.requestDatabaseFullSync(ODistributedAbstractPlugin.java:1493)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.requestFullDatabase(ODistributedAbstractPlugin.java:1269)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin$4.call(ODistributedAbstractPlugin.java:1100)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin$4.call(ODistributedAbstractPlugin.java:1006)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.executeInDistributedDatabaseLock(ODistributedAbstractPlugin.java:1916)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.installDatabase(ODistributedAbstractPlugin.java:1002)
at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl.validateStatus(ODistributedDatabaseImpl.java:1339)
at com.orientechnologies.orient.server.distributed.impl.task.OUpdateDatabaseSequenceStatusTask.execute(OUpdateDatabaseSequenceStatusTask.java:41)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin$2.call(ODistributedAbstractPlugin.java:686)
at com.orientechnologies.orient.core.db.OScenarioThreadLocal.executeAsDistributed(OScenarioThreadLocal.java:91)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.executeOnLocalNode(ODistributedAbstractPlugin.java:681)
at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl.execute(ODistributedDatabaseImpl.java:424)
at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl.lambda$processRequest$3(ODistributedDatabaseImpl.java:395)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.CancellationException
at java.util.concurrent.FutureTask.report(FutureTask.java:121)
at java.util.concurrent.FutureTask.get(FutureTask.java:206)
at com.orientechnologies.orient.core.db.viewmanager.ViewManager.close(ViewManager.java:177)
at com.orientechnologies.orient.server.distributed.impl.metadata.OSharedContextDistributed.close(OSharedContextDistributed.java:91)
at com.orientechnologies.orient.core.db.OrientDBEmbedded.networkRestore(OrientDBEmbedded.java:655)
... 23 more

2020-07-17 11:26:06:221 INFO [oriendb-cluster-1-demo-cloud-se-aws-us-east-1] - writing chunk #3 offset=0 size=8.00MB [OSyncReceiver]
2020-07-17 11:26:07:234 INFO [oriendb-cluster-1-demo-cloud-se-aws-us-east-1] - writing chunk #14 offset=0 size=8.00MB [OSyncReceiver]
2020-07-17 11:26:09:248 INFO [oriendb-cluster-1-demo-cloud-se-aws-us-east-1] - writing chunk #23 offset=0 size=8.00MB [OSyncReceiver]
2020-07-17 11:26:10:261 INFO [oriendb-cluster-1-demo-cloud-se-aws-us-east-1] - writing chunk #8 offset=0 size=8.00MB [OSyncReceiver]
2020-07-17 11:26:12:275 INFO [oriendb-cluster-1-demo-cloud-se-aws-us-east-1] - writing chunk #11 offset=0 size=8.00MB [OSyncReceiver]
2020-07-17 11:26:14:207 INFO [oriendb-cluster-1-demo-cloud-se-aws-us-east-1]->[[oriendb-cluster-2-demo-cloud-se-aws-us-east-1]] Requesting deploy of database 'xxx' on local server... [OHazelcastPlugin]
2020-07-17 11:26:14:288 INFO [oriendb-cluster-1-demo-cloud-se-aws-us-east-1] - writing chunk #6 offset=0 size=8.00MB [OSyncReceiver]
2020-07-17 11:26:15:301 INFO [oriendb-cluster-1-demo-cloud-se-aws-us-east-1] - writing chunk #2 offset=0 size=8.00MB [OSyncReceiver]
2020-07-17 11:26:17:313 INFO [oriendb-cluster-1-demo-cloud-se-aws-us-east-1] - writing chunk #4 offset=0 size=8.00MB [OSyncReceiver]
2020-07-17 11:26:18:326 INFO [oriendb-cluster-1-demo-cloud-se-aws-us-east-1] - writing chunk #15 offset=0 size=8.00MB [OSyncReceiver]
2020-07-17 11:26:20:338 INFO [oriendb-cluster-1-demo-cloud-se-aws-us-east-1] - writing chunk #24 offset=0 size=8.00MB [OSyncReceiver]

[lvca]

We fixed some timeout issues with a more recent version of 3.1.x. Please try it and in case the issue is still there, comment on this issue. Thanks.