Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ODB update failed #7776

Closed
weickdev opened this issue Sep 27, 2017 · 21 comments
Closed

ODB update failed #7776

weickdev opened this issue Sep 27, 2017 · 21 comments
Assignees

Comments

@weickdev
Copy link

OrientDB Version: 2.2.27

Java Version: 1.8

OS: Linux

I use the snapshot of 2.2.27 before, I update it to the newest release 2.2.27 version. Then I get those message:

2017-09-26 14:24:53:460 INFO {db=OSystem} Out of dated binary format for WAL segment /home/work/orientdb/orientdb-community-importers-2.2.27/databases/OSystem/OSystem.0.walin storageOSystemis detected, new segment will be added [ODiskWriteAheadLog] 2017-09-26 14:25:10:768 INFO {db=dev} Out of dated binary format for WAL segment/home/work/orientdb/orientdb-community-importers-2.2.27/databases/dev/dev.16083.walin storagedevis detected, new segment will be added [ODiskWriteAheadLog]

After that, there isn't any message. The Server is running, but I cannot find any databases. So how can I restore my datas?

@andrii0lomakin
Copy link
Member

@weickdev cold you list directories are located under databases on the server? What do you mean when you write that you cannot find any databases?

@weickdev
Copy link
Author

The directories under databases are:
dev GratefulDeadConcerts OSystem.

Under normal circumstances, I can see the dev and GratefulDeadConcerts database on the web studio's list as below.

database_list

But when I update to 2.2.27, the list is empty. Right now I have to reinstall the server, it works well now. Is it the same as #7756 ?

@andrii0lomakin
Copy link
Member

@weickdev no it is not related, otherwise, you would see an exception. Could you repeat all steps again? I mean use the previous snapshot, update distribution and check that list of databases is empty or not. I want to check whether the problem was gone or not.

@weickdev
Copy link
Author

@Laa OK, I will try both updating, from the snapshot to V2.2.27 and V2.2.28.

@weickdev
Copy link
Author

@Laa I update from V2.2.27 and V2.2.28, there are many logs like:
2017-09-27 16:41:03:291 SEVER {db=dev} Invalid WAL record type was passed class com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OFileTruncatedWALRecord. Given record will be skipped. [OLocalPaginatedStorage] 2017-09-27 16:41:03:291 SEVER {db=dev} Invalid WAL record type was passed class com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OFileTruncatedWALRecord. Given record will be skipped. [OLocalPaginatedStorage] 2017-09-27 16:41:03:292 SEVER {db=dev} Invalid WAL record type was passed class com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OFileTruncatedWALRecord. Given record will be skipped. [OLocalPaginatedStorage] 2017-09-27 16:41:03:292 SEVER {db=dev} Invalid WAL record type was passed class com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OFileTruncatedWALRecord. Given record will be skipped. [OLocalPaginatedStorage]

Does it matter?

@weickdev
Copy link
Author

@Laa I try to update to V2.2.27, I can login my database this time, but there are some exceptions:
Out of dated binary format for WAL segment/data/dfs/dn/1/orientdb/orientdb-community-importers-2.2.27/databases/dev/dev.16083.walin storagedevis detected, new segment will be added [ODiskWriteAheadLog]$ANSI{green {db=dev}} Exception4298D2FCin storagedevcom.orientechnologies.orient.core.exception.OConfigurationException: Cluster id 89 was not found in database 'dev' DB name="dev" at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.count(OAbstractPaginatedStorage.java:1259) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.countClusterElements(ODatabaseDocumentTx.java:820) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.countClusters(ONetworkProtocolBinary.java:866) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.executeRequest(ONetworkProtocolBinary.java:600) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.sessionRequest(ONetworkProtocolBinary.java:397) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:220) at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:77) $ANSI{green {db=dev}} Exception6AFD50E0in storagedevcom.orientechnologies.orient.core.exception.OConfigurationException: Cluster id 89 was not found in database 'dev' DB name="dev" at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.count(OAbstractPaginatedStorage.java:1259) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.countClusterElements(ODatabaseDocumentTx.java:820) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.countClusters(ONetworkProtocolBinary.java:866) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.executeRequest(ONetworkProtocolBinary.java:600) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.sessionRequest(ONetworkProtocolBinary.java:397) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:220) at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:77) $ANSI{green {db=dev}} Exception3159CE7Bin storagedevcom.orientechnologies.orient.core.exception.OConfigurationException: Cluster id 90 was not found in database 'dev' DB name="dev" at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.count(OAbstractPaginatedStorage.java:1259) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.countClusterElements(ODatabaseDocumentTx.java:820) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.countClusters(ONetworkProtocolBinary.java:866) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.executeRequest(ONetworkProtocolBinary.java:600) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.sessionRequest(ONetworkProtocolBinary.java:397) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:220) at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:77) $ANSI{green {db=dev}} Exception681CD3BCin storagedevcom.orientechnologies.orient.core.exception.OConfigurationException: Cluster id 73 was not found in database 'dev' DB name="dev" at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.count(OAbstractPaginatedStorage.java:1259) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.countClusterElements(ODatabaseDocumentTx.java:820) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.countClusters(ONetworkProtocolBinary.java:866) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.executeRequest(ONetworkProtocolBinary.java:600) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.sessionRequest(ONetworkProtocolBinary.java:397) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:220) at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:77) $ANSI{green {db=dev}} Exception28B51585in storagedevcom.orientechnologies.orient.core.exception.OConfigurationException: Cluster id 172 was not found in database 'dev' DB name="dev" at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.count(OAbstractPaginatedStorage.java:1259) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.countClusterElements(ODatabaseDocumentTx.java:820) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.countClusters(ONetworkProtocolBinary.java:866) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.executeRequest(ONetworkProtocolBinary.java:600) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.sessionRequest(ONetworkProtocolBinary.java:397) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:220) at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:77) $ANSI{green {db=dev}} Exception654E155Ein storagedevcom.orientechnologies.orient.core.exception.OConfigurationException: Cluster id 132 was not found in database 'dev' DB name="dev" at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.count(OAbstractPaginatedStorage.java:1259) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.countClusterElements(ODatabaseDocumentTx.java:820) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.countClusters(ONetworkProtocolBinary.java:866) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.executeRequest(ONetworkProtocolBinary.java:600) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.sessionRequest(ONetworkProtocolBinary.java:397) at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:220) at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:77)

@andrii0lomakin
Copy link
Member

@weickdev it means that you killed the server instead of soft shutdown. In general, we recommend do not kill servers during library updates. But this is interesting for me, data should be restored after the kill, could you send me initial database which you used for an upgrade?

@andrii0lomakin
Copy link
Member

@weickdev do you use any special settings or only default ones?

@weickdev
Copy link
Author

@Laa My Database is 90G. It is too huge that I cannot send it too you.

could you send me initial database which you used for an upgrade

@weickdev
Copy link
Author

@Laa There isn't any special settings. I only change the JVM parameters

do you use any special settings or only default ones?

@andrii0lomakin
Copy link
Member

@weickdev OK. Yes DB is too big. Could you then send me the content of all logs?

@weickdev
Copy link
Author

@Laa Which one do you want? The log updating to V2.2.27 or V2.2.28?

@andrii0lomakin
Copy link
Member

@weickdev I am a bit confused how did you make updates. Did you update to 2.2.27 and then after that to 2.2.28 or you updated from the initial database to 2.2.27 but then from the same copy of the initial database to 2.2.28?

@weickdev
Copy link
Author

@Laa Update from the initial database to 2.2.27, then from the same copy of the initial database to 2.2.28.

@andrii0lomakin
Copy link
Member

@weickdev OK, could you send both the logs then.

@weickdev
Copy link
Author

@Laa OK. But I am running batch edges inserting test, The log was overwritten. I will redo the updating when I finish my test. It may cost 2~3 days.

@weickdev
Copy link
Author

@Laa ,today I move the whole orientdb directory to another one, when I start the server, everything is ok. But if I try to connect db, I get this error:

2017-10-12 18:50:38:567 INFO  Loading configuration from: /data/dfs/dn/9/orientdb/orientdb-community-importers-2.2.28/config/orientdb-server-config.xml... [OServerConfigurationLoaderXml]
2017-10-12 18:50:38:823 INFO  OrientDB Server v2.2.28 (build 7f8db5be4bbc6b9f9382fdcce95c505c0588f238) is starting up... [OServer]
2017-10-12 18:50:38:836 INFO  Databases directory: /data/dfs/dn/9/orientdb/orientdb-community-importers-2.2.28/databases [OServer]
2017-10-12 18:50:38:909 INFO  OrientDB auto-config DISKCACHE=81,920MB (heap=7,851MB direct=81,920MB os=128,662MB) [OMemoryAndLocalPaginatedEnginesInitializer]
2017-10-12 18:50:38:910 INFO  Lowering disk cache size from 81,920MB to 81,918MB. [OGlobalConfiguration]
2017-10-12 18:50:39:217 INFO  Listening binary connections on 0.0.0.0:2424 (protocol v.36, socket=default) [OServerNetworkListener]
2017-10-12 18:50:39:221 INFO  Listening http connections on 0.0.0.0:2480 (protocol v.10, socket=default) [OServerNetworkListener]
2017-10-12 18:50:39:229 INFO  Installing dynamic plugin 'orientdb-teleporter-2.2.28.jar'... [OServerPluginManager]
2017-10-12 18:50:39:237 INFO  Installing dynamic plugin 'orientdb-studio-2.2.28.zip'... [OServerPluginManager]
2017-10-12 18:50:39:238 INFO  Installing dynamic plugin 'orientdb-neo4j-importer-plugin-2.2.28-dist.jar'... [OServerPluginManager]
2017-10-12 18:50:39:241 INFO  Installing dynamic plugin 'orientdb-etl-2.2.28.jar'... [OServerPluginManager]
2017-10-12 18:50:39:245 INFO  ODefaultPasswordAuthenticator is active [ODefaultPasswordAuthenticator]
2017-10-12 18:50:39:246 INFO  OServerConfigAuthenticator is active [OServerConfigAuthenticator]
2017-10-12 18:50:39:246 INFO  OSystemUserAuthenticator is active [OSystemUserAuthenticator]
2017-10-12 18:50:39:260 INFO  Installed GREMLIN language v.2.6.0 - graph.pool.max=50 [OGraphServerHandler]
2017-10-12 18:50:39:261 INFO  [OVariableParser.resolveVariables] Error on resolving property: distributed [orientechnologies]
2017-10-12 18:50:39:264 WARNI Authenticated clients can execute any kind of code into the server by using the following allowed languages: [sql] [OServerSideScriptInterpreter]
2017-10-12 18:50:39:265 INFO  OrientDB Studio available at http://192.168.1.20:2480/studio/index.html [OServer]
2017-10-12 18:50:39:266 INFO  OrientDB Server is active v2.2.28 (build 7f8db5be4bbc6b9f9382fdcce95c505c0588f238). [OServer]$ANSI{green {db=dev}} Exception `235E49C9` in storage `dev`
com.orientechnologies.orient.core.exception.OStorageException: Cannot open local storage '/data/dfs/dn/9/orientdb/orientdb-community-importers-2.2.28/databases/dev' with mode=rw
	DB name="dev"
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:304)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.open(ODatabaseDocumentTx.java:258)
	at com.orientechnologies.orient.server.OServer.openDatabase(OServer.java:977)
	at com.orientechnologies.orient.server.OServer.openDatabase(OServer.java:947)
	at com.orientechnologies.orient.server.OServer.openDatabase(OServer.java:938)
	at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.openDatabase(ONetworkProtocolBinary.java:913)
	at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.handshakeRequest(ONetworkProtocolBinary.java:251)
	at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:216)
	at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:81)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 2442
	at com.orientechnologies.orient.core.config.OStorageConfiguration.fromStream(OStorageConfiguration.java:508)
	at com.orientechnologies.orient.core.storage.impl.local.OStorageConfigurationSegment.load(OStorageConfigurationSegment.java:138)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:248)
	... 8 more

@andrii0lomakin
Copy link
Member

@weickdev I suppose you do not have a copy of an original database, do not you? Could you send me logs which you have?

@weickdev
Copy link
Author

I had copied all the files under orientdb-community-importers-2.2.28 directory. And databses is under orientdb-community-importers-2.2.28 directory. But now I am re importing all datas, so the logs has been delete. I will retry the moving after the datas are imported.

@andrii0lomakin
Copy link
Member

@weickdev could you provide information to understand this issue?

@andrii0lomakin
Copy link
Member

Close issue because of no reply. If you reproduce given issue and have information which is requested in issue feel free to reopen it.

@luigidellaquila luigidellaquila removed this from the 2.2.x (next hotfix) milestone Feb 5, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

4 participants