Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Synchronization in a cluster fails #7073

Closed
1 of 5 tasks
pavlop-asg opened this issue Jan 12, 2017 · 7 comments
Closed
1 of 5 tasks

Synchronization in a cluster fails #7073

pavlop-asg opened this issue Jan 12, 2017 · 7 comments
Assignees
@lvca
Labels
bug
Milestone
2.2.16

Comments

@pavlop-asg
Copy link

pavlop-asg commented Jan 12, 2017 •
edited
Loading

OrientDB Version, operating system, or hardware.

2.2.15-SNAPSHOT

Operating System

  • Linux
  • MacOSX
  • Windows
  • Other Unix
  • Other, name?

Expected behavior and actual behavior

This issue I have faced when I tested the issue #6999 after I was fixed.

There is an example. We have node-1 and after node-2, we write a value "1" to a document when node-1 and node-2 are up. We shut down node-2 and write value "2", after we shut down node-1 as well. When we start node-2 and after node-1 data is being synchronized, but we get the following issues:

  • I see strange logs which tell that node-1 does not see node-2, although replication works fine.
  • When I restart node-1 it does not join the cluster and does not see node-2.

Only when we restart all the cluster everything start working fine.

This are the logs from node-1:

2017-01-12 16:08:46.982 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.mobius.content.rest.api.config.StorageConfiguration$$EnhancerBySpringCGLIB$$413377f4:startEmbeddedOrientDbServerIfConfigured - Starting embedded OrientDB database...
2017-01-12 16:08:47.858 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.orientechnologies.orient.server.config.OServerConfigurationLoaderXml - Loading configuration from input stream
2017-01-12 16:08:49.084 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.orientechnologies.orient.server.OServer - OrientDB Server v2.2.15-SNAPSHOT is starting up...
2017-01-12 16:08:49.105 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.orientechnologies.orient.server.OServer - Databases directory: C:\Users\ppoh\asg\mobius\data\orientdb\databases
2017-01-12 16:08:49.252 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.orientechnologies.orient.core.engine.OMemoryAndLocalPaginatedEnginesInitializer - OrientDB auto-config DISKCACHE=3,614MB (heap=3,614MB direct=3,614MB os=16,260MB), assuming maximum direct memory size equals to maximum JVM heap size
2017-01-12 16:08:49.626 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.orientechnologies.orient.server.network.OServerNetworkListener - Listening $ANSI{green binary} connections on $ANSI{green 0.0.0.0:2424} (protocol v.36, socket=default)
2017-01-12 16:08:49.636 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.orientechnologies.orient.server.network.OServerNetworkListener - Listening $ANSI{green http} connections on $ANSI{green 0.0.0.0:2480} (protocol v.10, socket=default)
2017-01-12 16:08:49.656 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.orientechnologies.orient.server.plugin.OServerPluginManager - Installing dynamic plugin 'studio-2.2.zip'...
2017-01-12 16:08:49.666 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.orientechnologies.orient.server.security.authenticator.ODefaultPasswordAuthenticator - ODefaultPasswordAuthenticator is active
2017-01-12 16:08:49.676 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.orientechnologies.orient.server.security.authenticator.OServerConfigAuthenticator - OServerConfigAuthenticator is active
2017-01-12 16:08:49.676 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.orientechnologies.orient.server.security.authenticator.OSystemUserAuthenticator - OSystemUserAuthenticator is active
2017-01-12 16:08:49.716 [WARN ]    [RMI TCP Connection(3)-127.0.0.1] com.orientechnologies.orient.graph.gremlin.OGremlinHelper - GREMLIN language not available (not in classpath)
2017-01-12 16:08:51.813 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - Found 'replicator' user. Starting from OrientDB v2.2 this internal user is no needed anymore. Removing it...
2017-01-12 16:08:51.813 [WARN ]    [RMI TCP Connection(3)-127.0.0.1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - Assigning distributed node name: node-1
2017-01-12 16:08:51.933 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.hazelcast.config.FileSystemXmlConfig - Configuring Hazelcast from 'C:\Users\ppoh\asg\mobius\config\orientdb\hazelcast.xml'.
2017-01-12 16:08:52.797 [WARN ]    [RMI TCP Connection(3)-127.0.0.1] com.hazelcast.config.AbstractXmlConfigHelper - Name of the hazelcast schema location incorrect using default
2017-01-12 16:08:53.651 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.hazelcast.instance.DefaultAddressPicker - [LOCAL] [orientdb] [3.7.3] Prefer IPv4 stack is true.
2017-01-12 16:08:53.780 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.hazelcast.instance.DefaultAddressPicker - [LOCAL] [orientdb] [3.7.3] Picked [10.129.235.254]:2434, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=2434], bind any local is true
2017-01-12 16:08:53.837 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.hazelcast.system - [10.129.235.254]:2434 [orientdb] [3.7.3] Hazelcast 3.7.3 (20161117 - 8166eea) starting at [10.129.235.254]:2434
2017-01-12 16:08:53.847 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.hazelcast.system - [10.129.235.254]:2434 [orientdb] [3.7.3] Copyright (c) 2008-2016, Hazelcast, Inc. All Rights Reserved.
2017-01-12 16:08:53.847 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.hazelcast.system - [10.129.235.254]:2434 [orientdb] [3.7.3] Configured Hazelcast Serialization version : 1
2017-01-12 16:08:54.718 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.hazelcast.spi.impl.operationservice.impl.BackpressureRegulator - [10.129.235.254]:2434 [orientdb] [3.7.3] Backpressure is disabled
2017-01-12 16:08:57.205 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.hazelcast.instance.Node - [10.129.235.254]:2434 [orientdb] [3.7.3] Creating MulticastJoiner
2017-01-12 16:08:57.222 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.hazelcast.core.LifecycleService - [10.129.235.254]:2434 [orientdb] [3.7.3] [10.129.235.254]:2434 is STARTING
2017-01-12 16:08:57.790 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl - [10.129.235.254]:2434 [orientdb] [3.7.3] Starting 1 partition threads
2017-01-12 16:08:57.790 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl - [10.129.235.254]:2434 [orientdb] [3.7.3] Starting 2 generic threads (1 dedicated for priority tasks)
2017-01-12 16:08:57.800 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.hazelcast.nio.tcp.nonblocking.NonBlockingIOThreadingModel - [10.129.235.254]:2434 [orientdb] [3.7.3] TcpIpConnectionManager configured with Non Blocking IO-threading model: 3 input threads and 3 output threads
2017-01-12 16:09:00.087 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.hazelcast.internal.cluster.impl.MulticastJoiner - [10.129.235.254]:2434 [orientdb] [3.7.3] 


Members [1] {
	Member [10.129.235.254]:2434 - 4f072f95-a4d7-430f-8503-5a2e86b22e2d this
}

2017-01-12 16:09:00.227 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.hazelcast.core.LifecycleService - [10.129.235.254]:2434 [orientdb] [3.7.3] [10.129.235.254]:2434 is STARTED
2017-01-12 16:09:00.227 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - Starting distributed server 'node-1' (hzID=4f072f95-a4d7-430f-8503-5a2e86b22e2d)...
2017-01-12 16:09:00.227 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - Distributed cluster time=Thu Jan 12 16:09:00 EET 2017 (delta from local node=0)...
2017-01-12 16:09:00.425 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.hazelcast.internal.partition.impl.PartitionStateManager - [10.129.235.254]:2434 [orientdb] [3.7.3] Initializing cluster partition table arrangement...
2017-01-12 16:09:00.580 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - Distributed coordinator server is 'node-1'
2017-01-12 16:09:00.640 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.orientechnologies.orient.server.distributed.impl.ODistributedStorage - [node-1] Loaded configuration for database 'OSystem' from disk: C:\Users\ppoh\asg\mobius\config\orientdb\default-distributed-db-config.json
2017-01-12 16:09:00.660 [WARN ]    [RMI TCP Connection(3)-127.0.0.1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1] Servers in cluster: [node-1]
2017-01-12 16:09:01.310 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1] Opening database 'mobius'...
2017-01-12 16:09:01.320 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.orientechnologies.orient.server.distributed.impl.ODistributedStorage - [node-1] Loaded configuration for database 'mobius' from disk: C:\Users\ppoh\asg\mobius\data\orientdb\databases\mobius\distributed-config.json
2017-01-12 16:09:01.340 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1] Distributed servers status:

+-------+--------+---------+-----+---------+----------------+----------------+-------------------------+
|Name   |Status  |Databases|Conns|StartedOn|Binary          |HTTP            |UsedMemory               |
+-------+--------+---------+-----+---------+----------------+----------------+-------------------------+
|node-1*|STARTING|         |0    |16:08:49 |10.33.77.70:2424|10.33.77.70:2480|1010.01MB/3.53GB (27.94%)|
+-------+--------+---------+-----+---------+----------------+----------------+-------------------------+

2017-01-12 16:09:01.340 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin$4 - [node-1] Current node started as MASTER for database 'mobius'
2017-01-12 16:09:01.735 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1] Reassigning ownership of clusters for database mobius...
2017-01-12 16:09:01.735 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1] Reassignment of clusters for database 'mobius' completed (classes=22)
2017-01-12 16:09:01.759 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl - [node-1] Publishing ONLINE status for database node-1.mobius...
2017-01-12 16:09:01.760 [WARN ]    [RMI TCP Connection(3)-127.0.0.1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1] Updated node status to 'ONLINE'
2017-01-12 16:09:01.760 [WARN ]    [RMI TCP Connection(3)-127.0.0.1] com.orientechnologies.orient.server.handler.OServerSideScriptInterpreter - Authenticated clients can execute any kind of code into the server by using the following allowed languages: [sql]
2017-01-12 16:09:01.949 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.orientechnologies.orient.server.OServer - OrientDB Studio available at $ANSI{blue http://10.33.77.70:2480/studio/index.html}
2017-01-12 16:09:01.949 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.orientechnologies.orient.server.OServer - $ANSI{green:italic OrientDB Server is active} v2.2.15-SNAPSHOT.
2017-01-12 16:09:02.032 [INFO ]    [OrientDB (/127.0.0.1:2424) <- BinaryClient (/127.0.0.1:55948)] com.orientechnologies.orient.server.OClientConnectionManager - Remote client connected from: OClientConnection [id=1, source=/127.0.0.1:55948, since=1484230142032]
2017-01-12 16:09:02.145 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.hazelcast.instance.DefaultAddressPicker:log - [LOCAL] [dev] [3.7.3] Prefer IPv4 stack is true.
2017-01-12 16:09:02.223 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.hazelcast.instance.DefaultAddressPicker:log - [LOCAL] [dev] [3.7.3] Picked [10.129.235.254]:5701, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=5701], bind any local is true
2017-01-12 16:09:02.244 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.hazelcast.system:log - [10.129.235.254]:5701 [dev] [3.7.3] Hazelcast 3.7.3 (20161117 - 8166eea) starting at [10.129.235.254]:5701
2017-01-12 16:09:02.244 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.hazelcast.system:log - [10.129.235.254]:5701 [dev] [3.7.3] Copyright (c) 2008-2016, Hazelcast, Inc. All Rights Reserved.
2017-01-12 16:09:02.244 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.hazelcast.system:log - [10.129.235.254]:5701 [dev] [3.7.3] Configured Hazelcast Serialization version : 1
2017-01-12 16:09:02.266 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.hazelcast.spi.impl.operationservice.impl.BackpressureRegulator:log - [10.129.235.254]:5701 [dev] [3.7.3] Backpressure is disabled
2017-01-12 16:09:02.570 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.hazelcast.instance.Node:log - [10.129.235.254]:5701 [dev] [3.7.3] Creating MulticastJoiner
2017-01-12 16:09:02.580 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.hazelcast.core.LifecycleService:log - [10.129.235.254]:5701 [dev] [3.7.3] [10.129.235.254]:5701 is STARTING
2017-01-12 16:09:02.610 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl:log - [10.129.235.254]:5701 [dev] [3.7.3] Starting 4 partition threads
2017-01-12 16:09:02.620 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl:log - [10.129.235.254]:5701 [dev] [3.7.3] Starting 3 generic threads (1 dedicated for priority tasks)
2017-01-12 16:09:02.640 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.hazelcast.nio.tcp.nonblocking.NonBlockingIOThreadingModel:log - [10.129.235.254]:5701 [dev] [3.7.3] TcpIpConnectionManager configured with Non Blocking IO-threading model: 3 input threads and 3 output threads
2017-01-12 16:09:04.836 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.hazelcast.internal.cluster.impl.MulticastJoiner:log - [10.129.235.254]:5701 [dev] [3.7.3] 


Members [1] {
	Member [10.129.235.254]:5701 - 932f08b9-4e83-4908-90da-c8fccff9c99a this
}

2017-01-12 16:09:04.863 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.hazelcast.core.LifecycleService:log - [10.129.235.254]:5701 [dev] [3.7.3] [10.129.235.254]:5701 is STARTED
2017-01-12 16:09:05.471 [INFO ]    [OrientDB (/127.0.0.1:2424) <- BinaryClient (/127.0.0.1:55948)] com.orientechnologies.orient.server.OClientConnectionManager - $ANSI{green {db=mobius}} Remote client connected from: OClientConnection [id=2, source=/127.0.0.1:55948, since=1484230145471]
2017-01-12 16:09:05.501 [INFO ]    [OrientDB (/127.0.0.1:2424) <- BinaryClient (/127.0.0.1:55948)] com.orientechnologies.orient.server.OClientConnectionManager - Remote client connected from: OClientConnection [id=2, source=/127.0.0.1:55948, since=1484230145471]
2017-01-12 16:09:05.911 [INFO ]    [RMI TCP Connection(3)-127.0.0.1] com.hazelcast.internal.partition.impl.PartitionStateManager:log - [10.129.235.254]:5701 [dev] [3.7.3] Initializing cluster partition table arrangement...
2017-01-12 16:09:07.071 [INFO ]    [OrientDB (/127.0.0.1:2424) <- BinaryClient (/127.0.0.1:55948)] com.orientechnologies.orient.server.OClientConnectionManager - Remote client connected from: OClientConnection [id=3, source=/127.0.0.1:55948, since=1484230147071]
2017-01-12 16:09:07.071 [INFO ]    [OrientDB (/127.0.0.1:2424) <- BinaryClient (/127.0.0.1:55948)] com.orientechnologies.orient.server.OClientConnectionManager - Remote client connected from: OClientConnection [id=3, source=/127.0.0.1:55948, since=1484230147071]
2017-01-12 16:09:07.081 [INFO ]    [OrientDB (/127.0.0.1:2424) <- BinaryClient (/127.0.0.1:55948)] com.orientechnologies.orient.server.OClientConnectionManager - Remote client connected from: OClientConnection [id=4, source=/127.0.0.1:55948, since=1484230147081]
2017-01-12 16:09:07.091 [INFO ]    [OrientDB (/127.0.0.1:2424) <- BinaryClient (/127.0.0.1:55948)] com.orientechnologies.orient.server.OClientConnectionManager - Remote client connected from: OClientConnection [id=4, source=/127.0.0.1:55948, since=1484230147081]
2017-01-12 16:09:07.101 [INFO ]    [OrientDB (/127.0.0.1:2424) <- BinaryClient (/127.0.0.1:55967)] com.orientechnologies.orient.server.OClientConnectionManager - Remote client connected from: OClientConnection [id=5, source=/127.0.0.1:55967, since=1484230147101]
2017-01-12 16:09:07.102 [INFO ]    [OrientDB (/127.0.0.1:2424) <- BinaryClient (/127.0.0.1:55967)] com.orientechnologies.orient.server.OClientConnectionManager - Remote client connected from: OClientConnection [id=5, source=/127.0.0.1:55967, since=1484230147101]
2017-01-12 16:09:07.348 [INFO ]    [hz._hzInstance_1_orientdb.event-1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1] Distributed servers status:

+-------+------+----------------------+-----+---------+----------------+----------------+------------------------+
|Name   |Status|Databases             |Conns|StartedOn|Binary          |HTTP            |UsedMemory              |
+-------+------+----------------------+-----+---------+----------------+----------------+------------------------+
|node-1*|ONLINE|mobius=ONLINE (MASTER)|1    |16:08:49 |10.33.77.70:2424|10.33.77.70:2480|523.00MB/3.53GB (14.47%)|
+-------+------+----------------------+-----+---------+----------------+----------------+------------------------+

2017-01-12 16:11:32.176 [INFO ]    [hz._hzInstance_1_orientdb.IO.thread-Acceptor] com.hazelcast.nio.tcp.SocketAcceptorThread - [10.129.235.254]:2434 [orientdb] [3.7.3] Accepting socket connection from /10.129.235.254:56068
2017-01-12 16:11:32.187 [INFO ]    [hz._hzInstance_1_orientdb.cached.thread-3] com.hazelcast.nio.tcp.TcpIpConnectionManager - [10.129.235.254]:2434 [orientdb] [3.7.3] Established socket connection between /10.129.235.254:2434 and /10.129.235.254:56068
2017-01-12 16:11:39.181 [INFO ]    [hz._hzInstance_1_orientdb.priority-generic-operation.thread-0] com.hazelcast.internal.cluster.ClusterService - [10.129.235.254]:2434 [orientdb] [3.7.3] 

Members [2] {
	Member [10.129.235.254]:2434 - 4f072f95-a4d7-430f-8503-5a2e86b22e2d this
	Member [10.129.235.254]:2435 - 9a34fa8e-642a-4c6d-8376-905883b0bc72
}

2017-01-12 16:11:39.182 [WARN ]    [hz._hzInstance_1_orientdb.event-1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1] Added new node id=Member [10.129.235.254]:2435 - 9a34fa8e-642a-4c6d-8376-905883b0bc72 name=ext:9a34fa8e-642a-4c6d-8376-905883b0bc72
2017-01-12 16:11:39.425 [INFO ]    [hz._hzInstance_1_orientdb.migration] com.hazelcast.internal.partition.impl.MigrationManager - [10.129.235.254]:2434 [orientdb] [3.7.3] Re-partitioning cluster data... Migration queue size: 271
2017-01-12 16:11:41.228 [INFO ]    [hz._hzInstance_1_orientdb.migration] com.hazelcast.internal.partition.impl.MigrationThread - [10.129.235.254]:2434 [orientdb] [3.7.3] All migration tasks have been completed, queues are empty.
2017-01-12 16:11:41.343 [INFO ]    [hz._hzInstance_1_orientdb.event-1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1]<-[ext:9a34fa8e-642a-4c6d-8376-905883b0bc72] Received updated about registered nodes
2017-01-12 16:11:41.758 [INFO ]    [OrientDB (/10.33.77.70:2424) <- BinaryClient (/10.33.77.70:56075)] com.orientechnologies.orient.server.OClientConnectionManager - Remote client connected from: OClientConnection [id=6, source=/10.33.77.70:56075, since=1484230301758]
2017-01-12 16:11:41.761 [INFO ]    [OrientDB (/10.33.77.70:2424) <- BinaryClient (/10.33.77.70:56076)] com.orientechnologies.orient.server.OClientConnectionManager - Remote client connected from: OClientConnection [id=7, source=/10.33.77.70:56076, since=1484230301761]
2017-01-12 16:11:41.769 [INFO ]    [hz._hzInstance_1_orientdb.event-1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1]<-[node-2] Added node configuration id=Member [10.129.235.254]:2435 - 9a34fa8e-642a-4c6d-8376-905883b0bc72 name=node-2, now 2 nodes are configured
2017-01-12 16:11:41.771 [INFO ]    [hz._hzInstance_1_orientdb.event-1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1] Distributed servers status:

+-------+--------+----------------------+-----+---------+----------------+----------------+------------------------+
|Name   |Status  |Databases             |Conns|StartedOn|Binary          |HTTP            |UsedMemory              |
+-------+--------+----------------------+-----+---------+----------------+----------------+------------------------+
|node-1*|ONLINE  |mobius=ONLINE (MASTER)|4    |16:08:49 |10.33.77.70:2424|10.33.77.70:2480|847.51MB/3.53GB (23.45%)|
|node-2 |STARTING|                      |0    |16:11:27 |10.33.77.70:2425|10.33.77.70:2481|587.21MB/3.53GB (16.25%)|
+-------+--------+----------------------+-----+---------+----------------+----------------+------------------------+

2017-01-12 16:11:42.161 [INFO ]    [OrientDB DistributedWorker node=node-1 db=OSystem id=0] com.orientechnologies.orient.server.distributed.impl.ODistributedStorage - [node-1] Setting new distributed configuration for database: OSystem (version=1)

CLUSTER CONFIGURATION (LEGEND: X = Owner, o = Copy)
+--------+-----------+----------+
|CLUSTER |writeQuorum|readQuorum|
+--------+-----------+----------+
|*       |     1     |    1     |
|internal|     1     |    1     |
+--------+-----------+----------+


2017-01-12 16:11:42.243 [ERROR]    [OrientDB DistributedWorker node=node-1 db=mobius id=0] com.orientechnologies.orient.server.distributed.impl.task.OSyncDatabaseDeltaTask - [node-1]--[node-2] Skip deploying delta database 'mobius' because the requesting server has a most recent database (requester LastOperationOn=2017-01-12 16:07:43.280 current LastOperationOn=2017-01-12 16:05:12.621)
2017-01-12 16:11:42.259 [INFO ]    [hz._hzInstance_1_orientdb.event-1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1]<-[node-2] Received new status node-2.mobius=ONLINE
2017-01-12 16:11:42.261 [INFO ]    [hz._hzInstance_1_orientdb.event-1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1]<-[node-2] Received updated status node-1.mobius=NOT_AVAILABLE
2017-01-12 16:11:42.262 [INFO ]    [hz._hzInstance_1_orientdb.event-1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1] Distributed servers status:

+-------+--------+-----------------------------+-----+---------+----------------+----------------+------------------------+
|Name   |Status  |Databases                    |Conns|StartedOn|Binary          |HTTP            |UsedMemory              |
+-------+--------+-----------------------------+-----+---------+----------------+----------------+------------------------+
|node-1*|ONLINE  |mobius=NOT_AVAILABLE (MASTER)|6    |16:08:49 |10.33.77.70:2424|10.33.77.70:2480|911.90MB/3.53GB (25.23%)|
|node-2 |STARTING|                             |0    |16:11:27 |10.33.77.70:2425|10.33.77.70:2481|594.44MB/3.53GB (16.45%)|
+-------+--------+-----------------------------+-----+---------+----------------+----------------+------------------------+

2017-01-12 16:11:42.474 [INFO ]    [OrientDB (/127.0.0.1:2424) <- BinaryClient (/127.0.0.1:56078)] com.orientechnologies.orient.server.OClientConnectionManager - Remote client connected from: OClientConnection [id=8, source=/127.0.0.1:56078, since=1484230302474]
2017-01-12 16:11:42.520 [INFO ]    [Timer-1] com.orientechnologies.orient.server.distributed.impl.OClusterHealthChecker - [node-1] Trying to recover current server for database 'mobius'...
2017-01-12 16:11:42.535 [INFO ]    [Timer-1] com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin$3 - [node-1] Current node is a MASTER for database 'mobius'
2017-01-12 16:11:42.540 [WARN ]    [Timer-1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1]->[[node-2]] requesting delta database sync for 'mobius' on local server...
2017-01-12 16:11:42.541 [INFO ]    [Timer-1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1]->[node-2] Requesting database delta sync for 'mobius' LSN=LSN{segment=0, position=17829}...
2017-01-12 16:11:43.061 [INFO ]    [hz._hzInstance_1_orientdb.event-1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1]<-[node-2] Received updated status node-1.mobius=SYNCHRONIZING
2017-01-12 16:11:43.062 [INFO ]    [hz._hzInstance_1_orientdb.event-1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1] Distributed servers status:

+-------+--------+-----------------------------+-----+---------+----------------+----------------+------------------------+
|Name   |Status  |Databases                    |Conns|StartedOn|Binary          |HTTP            |UsedMemory              |
+-------+--------+-----------------------------+-----+---------+----------------+----------------+------------------------+
|node-1*|ONLINE  |mobius=SYNCHRONIZING (MASTER)|6    |16:08:49 |10.33.77.70:2424|10.33.77.70:2480|911.90MB/3.53GB (25.23%)|
|node-2 |STARTING|                             |0    |16:11:27 |10.33.77.70:2425|10.33.77.70:2481|594.44MB/3.53GB (16.45%)|
+-------+--------+-----------------------------+-----+---------+----------------+----------------+------------------------+

2017-01-12 16:11:43.091 [WARN ]    [Timer-1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1]<-[node-2] Error on installing database delta for 'mobius' (err=Requested database delta sync with LSN=LSN{segment=0, position=17829} but not found in database)
2017-01-12 16:11:43.091 [WARN ]    [Timer-1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1]<-[node-2] Requesting full database 'mobius' sync...
2017-01-12 16:11:43.092 [ERROR]    [Timer-1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1]->[node-2] Error on asking delta backup of database 'mobius' (err=Requested database delta sync with LSN=LSN{segment=0, position=17829} but not found in database)
2017-01-12 16:11:43.097 [INFO ]    [hz._hzInstance_2_dev.IO.thread-Acceptor] com.hazelcast.nio.tcp.SocketAcceptorThread:log - [10.129.235.254]:5701 [dev] [3.7.3] Accepting socket connection from /10.129.235.254:56093
2017-01-12 16:11:43.106 [INFO ]    [hz._hzInstance_2_dev.cached.thread-6] com.hazelcast.nio.tcp.TcpIpConnectionManager:log - [10.129.235.254]:5701 [dev] [3.7.3] Established socket connection between /10.129.235.254:5701 and /10.129.235.254:56093
2017-01-12 16:11:43.648 [WARN ]    [Timer-1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1]->[[node-2]] Requesting deploy of database 'mobius' on local server...
2017-01-12 16:11:43.650 [INFO ]    [Timer-1] com.orientechnologies.orient.server.distributed.impl.ODistributedStorage - [node-1] Downloaded configuration for database 'mobius' from the cluster
2017-01-12 16:11:43.656 [INFO ]    [Timer-1] com.orientechnologies.orient.server.distributed.impl.ODistributedStorage - [node-1] Setting new distributed configuration for database: mobius (version=1)

CLUSTER CONFIGURATION (LEGEND: X = Owner, o = Copy)
+---------------------+-----------+----------+-------------+-------+
|                     |           |          |   MASTER    |MASTER |
|                     |           |          |SYNCHRONIZING|ONLINE |
|                     |           |          |   dynamic   |dynamic|
+---------------------+-----------+----------+-------------+-------+
|CLUSTER              |writeQuorum|readQuorum|   node-1    |node-2 |
+---------------------+-----------+----------+-------------+-------+
|*                    |     1     |    1     |      X      |   o   |
|_studio              |     1     |    1     |      o      |   X   |
|_studio_1            |     1     |    1     |      o      |   X   |
|aclentity            |     1     |    1     |      o      |   X   |
|aclentity_1          |     1     |    1     |      o      |   X   |
|configurationentry   |     1     |    1     |      o      |   X   |
|configurationentry_3 |     1     |    1     |      o      |   X   |
|e                    |     1     |    1     |      o      |   X   |
|e_1                  |     1     |    1     |      o      |   X   |
|extractfilterentity_1|     1     |    1     |      o      |   X   |
|extractfilterentity_2|     1     |    1     |      o      |   X   |
|favoriteentity       |     1     |    1     |      o      |   X   |
|favoriteentity_3     |     1     |    1     |      o      |   X   |
|internal             |     1     |    1     |             |       |
|odocumentwrapper     |     1     |    1     |      o      |   X   |
|odocumentwrapper_2   |     1     |    1     |      o      |   X   |
|ofunction_0          |     1     |    1     |      o      |   X   |
|orole_0              |     1     |    1     |      o      |   X   |
|oschedule_0          |     1     |    1     |      o      |   X   |
|osequence_0          |     1     |    1     |      o      |   X   |
|ouser_0              |     1     |    1     |      o      |   X   |
|principalentity      |     1     |    1     |      o      |   X   |
|principalentity_1    |     1     |    1     |      o      |   X   |
|roleentity           |     1     |    1     |      o      |   X   |
|roleentity_1         |     1     |    1     |      o      |   X   |
|rulemodelentity_1    |     1     |    1     |      o      |   X   |
|rulemodelentity_3    |     1     |    1     |      o      |   X   |
|savedsearchentity    |     1     |    1     |      o      |   X   |
|savedsearchentity_3  |     1     |    1     |      o      |   X   |
|v                    |     1     |    1     |      o      |   X   |
|v_3                  |     1     |    1     |      o      |   X   |
+---------------------+-----------+----------+-------------+-------+


2017-01-12 16:11:43.672 [INFO ]    [hz._hzInstance_1_orientdb.event-1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1]<-[node-2] Received updated status node-2.mobius=BACKUP
2017-01-12 16:11:45.951 [INFO ]    [hz._hzInstance_1_orientdb.event-1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1]<-[node-2] Received updated status node-2.mobius=ONLINE
2017-01-12 16:11:46.110 [WARN ]    [Timer-1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1] Moving existent database 'mobius' in 'C:/Users/ppoh/asg/mobius/data/orientdb/databases/mobius' to 'C:/Users/ppoh/asg/mobius/data/orientdb/databases//../backup/databases/mobius' and get a fresh copy from a remote node...
2017-01-12 16:11:46.111 [ERROR]    [Timer-1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1] Error on moving existent database 'mobius' located in 'C:/Users/ppoh/asg/mobius/data/orientdb/databases/mobius' to 'C:\Users\ppoh\asg\mobius\data\orientdb\databases\..\backup\databases\mobius'. Deleting old database...
2017-01-12 16:11:46.115 [INFO ]    [Timer-1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1]<-[node-2] Copying remote database 'mobius' to: C:/Tomcats/tomcat_2443/temp/orientdb/install_mobius.zip
2017-01-12 16:11:46.141 [INFO ]    [Timer-1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1]<-[node-2] Installing database 'mobius' to: C:/Users/ppoh/asg/mobius/data/orientdb/databases/mobius...
2017-01-12 16:11:46.141 [INFO ]    [OrientDB installDatabase node=node-1 db=mobius] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1] - writing chunk #1 offset=0 size=389.02KB
2017-01-12 16:11:46.144 [INFO ]    [OrientDB installDatabase node=node-1 db=mobius] com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin$4 - [node-1] Database copied correctly, size=389.02KB
2017-01-12 16:11:47.699 [INFO ]    [hz._hzInstance_1_orientdb.event-1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1] Distributed servers status:

+-------+------+---------+-----+---------+----------------+----------------+------------------------+
|Name   |Status|Databases|Conns|StartedOn|Binary          |HTTP            |UsedMemory              |
+-------+------+---------+-----+---------+----------------+----------------+------------------------+
|node-1*|ONLINE|         |6    |16:08:49 |10.33.77.70:2424|10.33.77.70:2480|911.90MB/3.53GB (25.23%)|
|node-2 |ONLINE|         |2    |16:11:27 |10.33.77.70:2425|10.33.77.70:2481|748.65MB/3.53GB (20.71%)|
+-------+------+---------+-----+---------+----------------+----------------+------------------------+

2017-01-12 16:11:49.102 [INFO ]    [hz._hzInstance_2_dev.priority-generic-operation.thread-0] com.hazelcast.internal.cluster.ClusterService:log - [10.129.235.254]:5701 [dev] [3.7.3] 

Members [2] {
	Member [10.129.235.254]:5701 - 932f08b9-4e83-4908-90da-c8fccff9c99a this
	Member [10.129.235.254]:5702 - 060a097f-8d80-43d6-9df3-a26803e8d5b2
}

2017-01-12 16:11:49.102 [DEBUG]    [hz._hzInstance_2_dev.event-2] com.mobius.content.rm.service.dispatchers.EventDispatcherDistributed:memberAdded - New server's node added: MembershipEvent {member=Member [10.129.235.254]:5702 - 060a097f-8d80-43d6-9df3-a26803e8d5b2,type=added}
2017-01-12 16:11:49.150 [INFO ]    [OrientDB (/127.0.0.1:2424) <- BinaryClient (/127.0.0.1:56078)] com.orientechnologies.orient.server.OClientConnectionManager - $ANSI{green {db=mobius}} Remote client connected from: OClientConnection [id=9, source=/127.0.0.1:56078, since=1484230309150]
2017-01-12 16:11:49.296 [WARN ]    [Timer-1] com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage - $ANSI{green {db=mobius}} Storage 'mobius' was not closed properly. Will try to recover from write ahead log
2017-01-12 16:11:49.297 [ERROR]    [Timer-1] com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage - $ANSI{green {db=mobius}} Restore is not possible because write ahead log is empty.
2017-01-12 16:11:49.297 [INFO ]    [Timer-1] com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage - $ANSI{green {db=mobius}} Storage data recover was completed
2017-01-12 16:11:49.586 [INFO ]    [OrientDB (/127.0.0.1:2424) <- BinaryClient (/127.0.0.1:56078)] com.orientechnologies.orient.server.distributed.impl.ODistributedStorage - $ANSI{green {db=mobius}} [node-1] Downloaded configuration for database 'mobius' from the cluster
2017-01-12 16:11:49.586 [INFO ]    [Timer-1] com.orientechnologies.orient.server.distributed.impl.ODistributedStorage - $ANSI{green {db=mobius}} [node-1] Downloaded configuration for database 'mobius' from the cluster
2017-01-12 16:11:49.591 [INFO ]    [OrientDB (/127.0.0.1:2424) <- BinaryClient (/127.0.0.1:56078)] com.orientechnologies.orient.server.distributed.impl.ODistributedStorage - $ANSI{green {db=mobius}} [node-1] Setting new distributed configuration for database: mobius (version=1)

CLUSTER CONFIGURATION (LEGEND: X = Owner, o = Copy)
+---------------------+----+-----------+----------+-------------+-------+
|                     |    |           |          |   MASTER    |MASTER |
|                     |    |           |          |SYNCHRONIZING|ONLINE |
|                     |    |           |          |   dynamic   |dynamic|
+---------------------+----+-----------+----------+-------------+-------+
|CLUSTER              |  id|writeQuorum|readQuorum|   node-1    |node-2 |
+---------------------+----+-----------+----------+-------------+-------+
|*                    |    |     1     |    1     |      X      |   o   |
|_studio              |  58|     1     |    1     |      o      |   X   |
|_studio_1            |  59|     1     |    1     |      o      |   X   |
|aclentity            |  33|     1     |    1     |      o      |   X   |
|aclentity_1          |  34|     1     |    1     |      o      |   X   |
|configurationentry   |  29|     1     |    1     |      o      |   X   |
|configurationentry_3 |  32|     1     |    1     |      o      |   X   |
|e                    |  13|     1     |    1     |      o      |   X   |
|e_1                  |  14|     1     |    1     |      o      |   X   |
|extractfilterentity_1|  42|     1     |    1     |      o      |   X   |
|extractfilterentity_2|  43|     1     |    1     |      o      |   X   |
|favoriteentity       |  45|     1     |    1     |      o      |   X   |
|favoriteentity_3     |  48|     1     |    1     |      o      |   X   |
|internal             |   0|     1     |    1     |             |       |
|odocumentwrapper     |  17|     1     |    1     |      o      |   X   |
|odocumentwrapper_2   |  19|     1     |    1     |      o      |   X   |
|ofunction_0          |  55|     1     |    1     |      o      |   X   |
|orole_0              |  56|     1     |    1     |      o      |   X   |
|oschedule_0          |  53|     1     |    1     |      o      |   X   |
|osequence_0          |  54|     1     |    1     |      o      |   X   |
|ouser_0              |  57|     1     |    1     |      o      |   X   |
|principalentity      |  25|     1     |    1     |      o      |   X   |
|principalentity_1    |  26|     1     |    1     |      o      |   X   |
|roleentity           |  21|     1     |    1     |      o      |   X   |
|roleentity_1         |  22|     1     |    1     |      o      |   X   |
|rulemodelentity_1    |  38|     1     |    1     |      o      |   X   |
|rulemodelentity_3    |  40|     1     |    1     |      o      |   X   |
|savedsearchentity    |  49|     1     |    1     |      o      |   X   |
|savedsearchentity_3  |  52|     1     |    1     |      o      |   X   |
|v                    |   9|     1     |    1     |      o      |   X   |
|v_3                  |  12|     1     |    1     |      o      |   X   |
+---------------------+----+-----------+----------+-------------+-------+


2017-01-12 16:11:49.597 [INFO ]    [OrientDB (/127.0.0.1:2424) <- BinaryClient (/127.0.0.1:56078)] com.orientechnologies.orient.server.OClientConnectionManager - Remote client connected from: OClientConnection [id=9, source=/127.0.0.1:56078, since=1484230309150]
2017-01-12 16:11:49.613 [INFO ]    [Timer-1] com.orientechnologies.orient.server.distributed.impl.ODistributedStorage - $ANSI{green {db=mobius}} [node-1] Setting new distributed configuration for database: mobius (version=1)

CLUSTER CONFIGURATION (LEGEND: X = Owner, o = Copy)
+---------------------+----+-----------+----------+-------------+-------+
|                     |    |           |          |   MASTER    |MASTER |
|                     |    |           |          |SYNCHRONIZING|ONLINE |
|                     |    |           |          |   dynamic   |dynamic|
+---------------------+----+-----------+----------+-------------+-------+
|CLUSTER              |  id|writeQuorum|readQuorum|   node-1    |node-2 |
+---------------------+----+-----------+----------+-------------+-------+
|*                    |    |     1     |    1     |      X      |   o   |
|_studio              |  58|     1     |    1     |      o      |   X   |
|_studio_1            |  59|     1     |    1     |      o      |   X   |
|aclentity            |  33|     1     |    1     |      o      |   X   |
|aclentity_1          |  34|     1     |    1     |      o      |   X   |
|configurationentry   |  29|     1     |    1     |      o      |   X   |
|configurationentry_3 |  32|     1     |    1     |      o      |   X   |
|e                    |  13|     1     |    1     |      o      |   X   |
|e_1                  |  14|     1     |    1     |      o      |   X   |
|extractfilterentity_1|  42|     1     |    1     |      o      |   X   |
|extractfilterentity_2|  43|     1     |    1     |      o      |   X   |
|favoriteentity       |  45|     1     |    1     |      o      |   X   |
|favoriteentity_3     |  48|     1     |    1     |      o      |   X   |
|internal             |   0|     1     |    1     |             |       |
|odocumentwrapper     |  17|     1     |    1     |      o      |   X   |
|odocumentwrapper_2   |  19|     1     |    1     |      o      |   X   |
|ofunction_0          |  55|     1     |    1     |      o      |   X   |
|orole_0              |  56|     1     |    1     |      o      |   X   |
|oschedule_0          |  53|     1     |    1     |      o      |   X   |
|osequence_0          |  54|     1     |    1     |      o      |   X   |
|ouser_0              |  57|     1     |    1     |      o      |   X   |
|principalentity      |  25|     1     |    1     |      o      |   X   |
|principalentity_1    |  26|     1     |    1     |      o      |   X   |
|roleentity           |  21|     1     |    1     |      o      |   X   |
|roleentity_1         |  22|     1     |    1     |      o      |   X   |
|rulemodelentity_1    |  38|     1     |    1     |      o      |   X   |
|rulemodelentity_3    |  40|     1     |    1     |      o      |   X   |
|savedsearchentity    |  49|     1     |    1     |      o      |   X   |
|savedsearchentity_3  |  52|     1     |    1     |      o      |   X   |
|v                    |   9|     1     |    1     |      o      |   X   |
|v_3                  |  12|     1     |    1     |      o      |   X   |
+---------------------+----+-----------+----------+-------------+-------+


2017-01-12 16:11:49.627 [INFO ]    [OrientDB (/127.0.0.1:2424) <- BinaryClient (/127.0.0.1:56078)] com.orientechnologies.orient.server.OClientConnectionManager - Remote client connected from: OClientConnection [id=10, source=/127.0.0.1:56078, since=1484230309626]
2017-01-12 16:11:50.422 [INFO ]    [OrientDB (/127.0.0.1:2424) <- BinaryClient (/127.0.0.1:56078)] com.orientechnologies.orient.server.OClientConnectionManager - Remote client connected from: OClientConnection [id=10, source=/127.0.0.1:56078, since=1484230309626]
2017-01-12 16:11:50.434 [INFO ]    [OrientDB (/127.0.0.1:2424) <- BinaryClient (/127.0.0.1:56078)] com.orientechnologies.orient.server.OClientConnectionManager - Remote client connected from: OClientConnection [id=11, source=/127.0.0.1:56078, since=1484230310434]
2017-01-12 16:11:50.435 [INFO ]    [OrientDB (/127.0.0.1:2424) <- BinaryClient (/127.0.0.1:56078)] com.orientechnologies.orient.server.OClientConnectionManager - Remote client connected from: OClientConnection [id=11, source=/127.0.0.1:56078, since=1484230310434]
2017-01-12 16:11:50.437 [INFO ]    [Timer-1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1] Installed database 'mobius' (LSN=LSN{segment=0, position=86})
2017-01-12 16:11:50.440 [INFO ]    [Timer-1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1] Reassigning ownership of clusters for database mobius...
2017-01-12 16:11:50.455 [INFO ]    [Timer-1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1] Reassignment of clusters for database 'mobius' completed (classes=22)
2017-01-12 16:11:50.456 [INFO ]    [Timer-1] com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl - [node-1] Publishing ONLINE status for database node-1.mobius...
2017-01-12 16:11:50.456 [INFO ]    [OrientDB (/127.0.0.1:2424) <- BinaryClient (/127.0.0.1:56078)] com.orientechnologies.orient.server.OClientConnectionManager - Remote client connected from: OClientConnection [id=12, source=/127.0.0.1:56078, since=1484230310456]
2017-01-12 16:11:50.457 [INFO ]    [OrientDB (/127.0.0.1:2424) <- BinaryClient (/127.0.0.1:56078)] com.orientechnologies.orient.server.OClientConnectionManager - Remote client connected from: OClientConnection [id=12, source=/127.0.0.1:56078, since=1484230310456]
2017-01-12 16:11:50.459 [INFO ]    [hz._hzInstance_1_orientdb.event-1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1] Received updated status node-1.mobius=ONLINE
2017-01-12 16:11:50.472 [INFO ]    [hz._hzInstance_1_orientdb.event-1] com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin - [node-1] Distributed servers status:

+-------+------+----------------------+-----+---------+----------------+----------------+------------------------+
|Name   |Status|Databases             |Conns|StartedOn|Binary          |HTTP            |UsedMemory              |
+-------+------+----------------------+-----+---------+----------------+----------------+------------------------+
|node-1*|ONLINE|mobius=ONLINE (MASTER)|6    |16:08:49 |10.33.77.70:2424|10.33.77.70:2480|911.90MB/3.53GB (25.23%)|
|node-2 |ONLINE|mobius=ONLINE (MASTER)|2    |16:11:27 |10.33.77.70:2425|10.33.77.70:2481|748.65MB/3.53GB (20.71%)|
+-------+------+----------------------+-----+---------+----------------+----------------+------------------------+

2017-01-12 16:11:50.484 [INFO ]    [Timer-1] com.orientechnologies.orient.server.distributed.impl.OClusterHealthChecker - [node-1] Recover complete for database 'mobius'...
2017-01-12 16:11:50.986 [INFO ]    [hz._hzInstance_2_dev.migration] com.hazelcast.internal.partition.impl.MigrationManager:log - [10.129.235.254]:5701 [dev] [3.7.3] Re-partitioning cluster data... Migration queue size: 271
2017-01-12 16:11:52.939 [INFO ]    [hz._hzInstance_2_dev.migration] com.hazelcast.internal.partition.impl.MigrationThread:log - [10.129.235.254]:5701 [dev] [3.7.3] All migration tasks have been completed, queues are empty.
2017-01-12 16:12:10.488 [WARN ]    [Timer-1] com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl - [node-1] Timeout (20002ms) on waiting for synchronous responses from nodes=[node-2] responsesSoFar=[] request=(id=0.2 task=heartbeat timestamp: 1484230310486)
2017-01-12 16:12:10.489 [WARN ]    [Timer-1] com.orientechnologies.orient.server.distributed.impl.OClusterHealthChecker - [node-1]->[node-2] Server 'node-2' did not respond to the heartbeat message (db=mobius, timeout=10000ms), but cannot be set OFFLINE by configuration
2017-01-12 16:12:20.797 [WARN ]    [Timer-1] com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl - [node-1] Timeout (10002ms) on waiting for synchronous responses from nodes=[node-2] responsesSoFar=[] request=(id=0.3 task=heartbeat timestamp: 1484230330795)
2017-01-12 16:12:20.797 [WARN ]    [Timer-1] com.orientechnologies.orient.server.distributed.impl.OClusterHealthChecker - [node-1]->[node-2] Server 'node-2' did not respond to the heartbeat message (db=mobius, timeout=10000ms), but cannot be set OFFLINE by configuration
2017-01-12 16:12:31.113 [WARN ]    [Timer-1] com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl - [node-1] Timeout (10001ms) on waiting for synchronous responses from nodes=[node-2] responsesSoFar=[] request=(id=0.4 task=heartbeat timestamp: 1484230341112)
2017-01-12 16:12:31.113 [WARN ]    [Timer-1] com.orientechnologies.orient.server.distributed.impl.OClusterHealthChecker - [node-1]->[node-2] Server 'node-2' did not respond to the heartbeat message (db=mobius, timeout=10000ms), but cannot be set OFFLINE by configuration
2017-01-12 16:12:41.432 [WARN ]    [Timer-1] com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl - [node-1] Timeout (10001ms) on waiting for synchronous responses from nodes=[node-2] responsesSoFar=[] request=(id=0.5 task=heartbeat timestamp: 1484230351431)
2017-01-12 16:12:41.432 [WARN ]    [Timer-1] com.orientechnologies.orient.server.distributed.impl.OClusterHealthChecker - [node-1]->[node-2] Server 'node-2' did not respond to the heartbeat message (db=mobius, timeout=10000ms), but cannot be set OFFLINE by configuration
2017-01-12 16:12:51.736 [WARN ]    [Timer-1] com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl - [node-1] Timeout (10002ms) on waiting for synchronous responses from nodes=[node-2] responsesSoFar=[] request=(id=0.6 task=heartbeat timestamp: 1484230361734)
2017-01-12 16:12:51.736 [WARN ]    [Timer-1] com.orientechnologies.orient.server.distributed.impl.OClusterHealthChecker - [node-1]->[node-2] Server 'node-2' did not respond to the heartbeat message (db=mobius, timeout=10000ms), but cannot be set OFFLINE by configuration
2017-01-12 16:13:02.047 [WARN ]    [Timer-1] com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl - [node-1] Timeout (10002ms) on waiting for synchronous responses from nodes=[node-2] responsesSoFar=[] request=(id=0.7 task=heartbeat timestamp: 1484230372045)
2017-01-12 16:13:02.047 [WARN ]    [Timer-1] com.orientechnologies.orient.server.distributed.impl.OClusterHealthChecker - [node-1]->[node-2] Server 'node-2' did not respond to the heartbeat message (db=mobius, timeout=10000ms), but cannot be set OFFLINE by configuration
2017-01-12 16:13:12.364 [WARN ]    [Timer-1] com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl - [node-1] Timeout (10001ms) on waiting for synchronous responses from nodes=[node-2] responsesSoFar=[] request=(id=0.8 task=heartbeat timestamp: 1484230382363)
2017-01-12 16:13:12.364 [WARN ]    [Timer-1] com.orientechnologies.orient.server.distributed.impl.OClusterHealthChecker - [node-1]->[node-2] Server 'node-2' did not respond to the heartbeat message (db=mobius, timeout=10000ms), but cannot be set OFFLINE by configuration
2017-01-12 16:13:22.676 [WARN ]    [Timer-1] com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl - [node-1] Timeout (10001ms) on waiting for synchronous responses from nodes=[node-2] responsesSoFar=[] request=(id=0.9 task=heartbeat timestamp: 1484230392675)
@pavlop-asg pavlop-asg mentioned this issue Jan 12, 2017
Closed
@lvca lvca self-assigned this Jan 14, 2017
@lvca lvca added the bug label Jan 14, 2017
@santo-it santo-it added this to the 2.2.x (next hotfix) milestone Jan 24, 2017
@lvca
Copy link
Member

lvca commented Jan 25, 2017

@pavlop-asg do you have the log of the other server too?

@lvca
Copy link
Member

lvca commented Jan 25, 2017

Could you please post the hazelcast.xml of both servers?

@pavlop-asg
Copy link
Author

They are the same for both

<?xml version="1.0" encoding="UTF-8"?>
<hazelcast
		xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-3.5.xsd"
		xmlns="http://www.hazelcast.com/schema/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
	<group>
		<name>orientdb</name>
		<password>orientdb</password>
	</group>
	<properties>
		<property name="hazelcast.mancenter.enabled">false</property>
		<property name="hazelcast.memcache.enabled">false</property>
		<property name="hazelcast.rest.enabled">false</property>
		<property name="hazelcast.wait.seconds.before.join">5</property>
		<property name="hazelcast.operation.thread.count">1</property>
		<property name="hazelcast.operation.generic.thread.count">1</property>
		<property name="hazelcast.client.event.thread.count">1</property>
		<property name="hazelcast.event.thread.count">1</property>

	</properties>
	<network>
		<port auto-increment="true">2434</port>
		<join>
			<multicast enabled="true">
				<multicast-group>235.1.1.1</multicast-group>
				<multicast-port>2494</multicast-port>
			</multicast>
		</join>
	</network>
	<executor-service>
		<pool-size>16</pool-size>
	</executor-service>
</hazelcast>

@pavlop-asg
Copy link
Author

Looks like your fix helped

@santo-it santo-it modified the milestones: 2.2.16, 2.2.x (next hotfix) Jan 25, 2017
@santo-it
Copy link

Hi @pavlop-asg,

Glad to hear the issue has been fixed. Thanks for your patience and tests
I'm going ahead and closing this issue. Feel free to reopen if anything

Thanks,

@santo-it
Copy link

Fixed by ce26174

Release notes updated

No changes in the documentation required

Closing

@340370
Copy link

340370 commented May 20, 2019 •
edited
Loading

What is the fix? Please share the details

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lvca lvca

Labels
bug
Milestone
2.2.16
Development

No branches or pull requests
4 participants
@lvca @pavlop-asg @santo-it @340370