Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error on creating ETL processor #7109

Closed
ahr12 opened this issue Jan 23, 2017 · 8 comments
Closed

Error on creating ETL processor #7109

ahr12 opened this issue Jan 23, 2017 · 8 comments
Assignees
@robfrank
Labels
bug waiting reply
Milestone
2.2.16

Comments

@ahr12
Copy link

ahr12 commented Jan 23, 2017

Hi, I'm trying to upload graph from JSON with JSON ETL.
System parameters: OrientDB 2.2.14, ubuntu 14.04 64 bit, JAVA 1.8

I am gating the followings errors -
Exception in thread "main" com.orientechnologies.orient.core.exception.OConfigurationException: Error on creating ETL processor
DB name="plocal:/tmp/databases/testdb"
at com.orientechnologies.orient.etl.OETLProcessor.parse(OETLProcessor.java:245)
at com.orientechnologies.orient.etl.OETLProcessor.parse(OETLProcessor.java:186)
at com.orientechnologies.orient.etl.OETLProcessor.parseConfigAndParameters(OETLProcessor.java:176)
at com.orientechnologies.orient.etl.OETLProcessor.main(OETLProcessor.java:140)
Caused by: java.lang.NullPointerException
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.get(ODatabaseDocumentTx.java:1468)
at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.readDatabaseConfiguration(OrientBaseGraph.java:1851)
at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.(OrientBaseGraph.java:146)
at com.tinkerpop.blueprints.impls.orient.OrientGraphNoTx.(OrientGraphNoTx.java:62)
at com.tinkerpop.blueprints.impls.orient.OrientGraphFactory$2.getGraph(OrientGraphFactory.java:116)
at com.tinkerpop.blueprints.impls.orient.OrientGraphFactory.getNoTx(OrientGraphFactory.java:240)
at com.orientechnologies.orient.etl.loader.OOrientDBLoader.configureGraphDB(OOrientDBLoader.java:407)
at com.orientechnologies.orient.etl.loader.OOrientDBLoader.configure(OOrientDBLoader.java:345)
at com.orientechnologies.orient.etl.OETLProcessor.configureComponent(OETLProcessor.java:484)
at com.orientechnologies.orient.etl.OETLProcessor.configureLoader(OETLProcessor.java:297)
at com.orientechnologies.orient.etl.OETLProcessor.parse(OETLProcessor.java:229)
... 3 more
@robfrank robfrank self-assigned this Jan 23, 2017
@robfrank
Copy link
Contributor

Could you provide the configuration Json and some sample data?

@ahr12
Copy link
Author

ahr12 commented Jan 23, 2017

configuration JSON file

{
"config": {
"log": "debug"
},
"source" : {
"file": { "path": "/tmp/test.json" }
},
"extractor" : {
"json": {}
},
"transformers" : [
{ "merge": { "joinFieldName": "id", "lookup": "Account.id" } },
{ "vertex": { "class": "Account"} },
{ "edge": {
"class": "Friend",
"joinFieldName": "friends",
"lookup": "Account.id",
"unresolvedLinkAction": "CREATE"
} },
{ "edge": {
"class": "Enemy",
"joinFieldName": "enemies",
"lookup": "Account.id",
"unresolvedLinkAction": "CREATE"
} }
],
"loader" : {
"orientdb": {
"dbURL": "plocal:/tmp/databases/testdb",
"dbUser": "admin",
"dbPassword": "admin",
"dbAutoDropIfExists": true,
"dbAutoCreate": true,
"standardElementConstraints": false,
"tx": false,
"wal": false,
"batchCommit": 1000,
"dbType": "graph",
"classes": [{"name": "Account", "extends":"V"}, {"name": "Friend", "extends":"E"}, {"name": 'Enemy', "extends":"E"}],
"indexes": [{"class":"Account", "fields":["id:integer"], "type":"UNIQUE_HASH_INDEX" }]
}
}
}

Sample data - test.json

[
{
"name": "Joe",
"id": 1,
"friends": [2,4,5],
"enemies": [6]
},
{
"name": "Suzie",
"id": 2,
"friends": [1,4,6],
"enemies": [5,2]
}
]

@saeedtabrizi
Copy link
Contributor

saeedtabrizi commented Jan 23, 2017 •
edited
Loading

Hi @robfrank , i have same problem in odb 2.2.15

osm data can take from OpenStreetMap as test.osm

configuration :
{ "config": { "log": "debug" }, "begin": [ { "let": { "name": "$filePath", "value": "6565" } }, { "let": { "name": "$className", "value": "hello" } } ], "source": { "file": { "path": "/mnt/hdd2/test.osm" } }, "extractor": { "xml": { "rootNode": "osm.node" } }, "transformers": [ { "vertex": { "class": "Osm_Node" } } ], "loader": { "output": {} } }

my output log :


 OrientDB etl v.2.2.15 (build 2.2.x@rebf7b80c08613e871633edeb39c5c985e26b176c; 2017-01-19
07:39:00+0000) www.orientdb.com
BEGIN ETL PROCESSOR
[file] INFO Reading from file /mnt/hdd2/test.osm with encoding UTF-8
Started execution with 1 worker threads
ETL process has problem: java.lang.NullPointerException
END ETL PROCESSOR
+ extracted 0 entries (0 entries/sec) - 0 entries -> loaded 0 bytes (0 bytes/sec) Total t
ime: 77ms [0 warnings, 0 errors]

but wen i use following configuration , everything is ok :

{
    "config": {
        "log": "debug"
    },
    "begin": [
        {
            "let": {
                "name": "$filePath",
                "value": "6565"
            }
        },
        {
            "let": {
                "name": "$className",
                "value": "hello"
            }
        }
    ],
    "source": {
        "file": {
            "path": "/mnt/hdd2/test.osm"
        }
    },
    "extractor": {
        "xml": {
            "rootNode": "osm.node"
        }
    },
    "transformers": [
        
    ],
    "loader": {
        "output": {}
    }
}

output log :


OrientDB etl v.2.2.15 (build 2.2.x@rebf7b80c08613e871633edeb39c5c985e26b176c; 2017-01-19 07:39:00+0000) www.orientdb.com
BEGIN ETL PROCESSOR
[file] INFO Reading from file /mnt/hdd2/test.osm with encoding UTF-8
Started execution with 1 worker threads
Start extracting
{changeset:45332095,id:4620980938,lat:35.7086218,lon:51.4109069,timestamp:2017-01-20T17:20:29Z,uid:3360520,user:madjidf,version:1,visible:true}
{changeset:45332039,id:4620979384,lat:35.7116648,lon:51.4105953,timestamp:2017-01-20T17:18:10Z,uid:3360520,user:madjidf,version:1,visible:true}
{changeset:45332039,id:4620979386,lat:35.7111156,lon:51.4106510,timestamp:2017-01-20T17:18:10Z,uid:3360520,user:madjidf,version:1,visible:true}
{changeset:45332039,id:4620979387,lat:35.7110695,lon:51.4100838,timestamp:2017-01-20T17:18:10Z,uid:3360520,user:madjidf,version:1,visible:true}
{changeset:45332039,id:4620979388,lat:35.7106872,lon:51.4106944,timestamp:2017-01-20T17:18:10Z,uid:3360520,user:madjidf,version:1,visible:true}
{changeset:45332039,id:4620980189,lat:35.7106470,lon:51.4100919,timestamp:2017-01-20T17:18:10Z,uid:3360520,user:madjidf,version:1,visible:true}
{changeset:45332039,id:4620980190,lat:35.7105810,lon:51.4095878,timestamp:2017-01-20T17:18:10Z,uid:3360520,user:madjidf,version:1,visible:true}
{changeset:45332039,id:4620980191,lat:35.7098157,lon:51.4107827,timestamp:2017-01-20T17:18:10Z,uid:3360520,user:madjidf,version:1,visible:true}
{changeset:45332039,id:4620980192,lat:35.7098283,lon:51.4112547,timestamp:2017-01-20T17:18:10Z,uid:3360520,user:madjidf,version:1,visible:true}
{changeset:45332039,id:4620980195,lat:35.7092527,lon:51.4108399,timestamp:2017-01-20T17:18:10Z,uid:3360520,user:madjidf,version:1,visible:true}
{changeset:42880693,id:2021254176,lat:35.7105896,lon:51.4092324,timestamp:2016-10-13T20:24:07Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:42880693,id:2021254177,lat:35.7112705,lon:51.4091419,timestamp:2016-10-13T20:24:07Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:42880693,id:2021254178,lat:35.7104456,lon:51.4068835,timestamp:2016-10-13T20:24:07Z,uid:4449060,user:Khalil Laleh,version:4,visible:true}
{changeset:42880693,id:2021254175,lat:35.7105020,lon:51.4083992,timestamp:2016-10-13T20:24:07Z,uid:4449060,user:Khalil Laleh,version:4,visible:true}
{changeset:35751999,id:3876630879,lat:35.7112496,lon:51.4088079,timestamp:2015-12-04T17:04:19Z,uid:25975,user:AmZaf,version:1,visible:true}
{changeset:35751999,id:3876630880,lat:35.7110050,lon:51.4088303,timestamp:2015-12-04T17:04:19Z,uid:25975,user:AmZaf,version:1,visible:true}
{changeset:35751999,id:3876630881,lat:35.7112280,lon:51.4084634,timestamp:2015-12-04T17:04:19Z,uid:25975,user:AmZaf,version:1,visible:true}
{changeset:35751999,id:3876630882,lat:35.7105069,lon:51.4084457,timestamp:2015-12-04T17:04:19Z,uid:25975,user:AmZaf,version:1,visible:true}
{changeset:35751999,id:3876630883,lat:35.7113056,lon:51.4097014,timestamp:2015-12-04T17:04:19Z,uid:25975,user:AmZaf,version:1,visible:true}
{changeset:35751999,id:3876630884,lat:35.7106065,lon:51.4097744,timestamp:2015-12-04T17:04:19Z,uid:25975,user:AmZaf,version:1,visible:true}
{changeset:36303811,id:3922337347,lat:35.7106629,lon:51.4081717,timestamp:2016-01-01T18:40:46Z,uid:3012890,user:HADiMozafari,version:1,visible:true}
{changeset:36303811,id:3922337348,lat:35.7106651,lon:51.4082522,timestamp:2016-01-01T18:40:46Z,uid:3012890,user:HADiMozafari,version:1,visible:true}
{changeset:36303811,id:3922337349,lat:35.7105148,lon:51.4082522,timestamp:2016-01-01T18:40:47Z,uid:3012890,user:HADiMozafari,version:1,visible:true}
{changeset:36303811,id:3922337350,lat:35.7104936,lon:51.4081726,timestamp:2016-01-01T18:40:47Z,uid:3012890,user:HADiMozafari,version:1,visible:true}
{changeset:38168129,id:4086609684,lat:35.7108022,lon:51.4084778,timestamp:2016-03-30T14:29:32Z,uid:3312986,user:Amir Ahmad Mohamadi,version:1,visible:true}
{changeset:38168129,id:4086609685,lat:35.7108057,lon:51.4087093,timestamp:2016-03-30T14:29:32Z,uid:3312986,user:Amir Ahmad Mohamadi,version:1,visible:true}
{changeset:38168129,id:4086609686,lat:35.7111975,lon:51.4087007,timestamp:2016-03-30T14:29:32Z,uid:3312986,user:Amir Ahmad Mohamadi,version:1,visible:true}
{changeset:38168129,id:4086609687,lat:35.7111852,lon:51.4084887,timestamp:2016-03-30T14:29:32Z,uid:3312986,user:Amir Ahmad Mohamadi,version:1,visible:true}
{changeset:42880322,id:1649546841,lat:35.7112249,lon:51.4072513,timestamp:2016-10-13T20:07:26Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:42880693,id:1730748508,lat:35.7086936,lon:51.4091692,timestamp:2016-10-13T20:13:28Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:42880693,id:1730748523,lat:35.7087626,lon:51.4092329,timestamp:2016-10-13T20:13:28Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:42880693,id:1730748528,lat:35.7088236,lon:51.4108834,timestamp:2016-10-13T20:13:28Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:42880693,id:1730748534,lat:35.7088531,lon:51.4107725,timestamp:2016-10-13T20:13:28Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:42880693,id:1730748574,lat:35.7093578,lon:51.4091232,timestamp:2016-10-13T20:13:28Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:42880693,id:1730748594,lat:35.7098728,lon:51.4067574,timestamp:2016-10-13T20:13:29Z,uid:4449060,user:Khalil Laleh,version:4,visible:true}
{changeset:42880693,id:1730748608,lat:35.7099457,lon:51.4091273,timestamp:2016-10-13T20:13:29Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:42880693,id:1730748628,lat:35.7100022,lon:51.4090786,timestamp:2016-10-13T20:13:29Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:42880693,id:1730748632,lat:35.7100362,lon:51.4106669,timestamp:2016-10-13T20:13:29Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:42880693,id:1730748634,lat:35.7100960,lon:51.4107543,timestamp:2016-10-13T20:13:29Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:42880693,id:1730748641,lat:35.7101567,lon:51.4117777,timestamp:2016-10-13T20:13:29Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:44853153,id:1730748428,lat:35.7081525,lon:51.4109616,timestamp:2017-01-02T19:16:20Z,uid:3360520,user:madjidf,version:3,visible:true}
{changeset:42880693,id:1730748734,lat:35.7111627,lon:51.4074215,timestamp:2016-10-13T20:13:29Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:42880693,id:1730748748,lat:35.7113635,lon:51.4106259,timestamp:2016-10-13T20:13:29Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:42880693,id:1730748800,lat:35.7120109,lon:51.4105602,timestamp:2016-10-13T20:13:29Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:42880693,id:1730748814,lat:35.7122042,lon:51.4105406,timestamp:2016-10-13T20:13:29Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:42880693,id:1730748893,lat:35.7127288,lon:51.4104874,timestamp:2016-10-13T20:13:30Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:44340601,id:4456355194,lat:35.7108694,lon:51.4109623,timestamp:2016-12-12T09:21:40Z,uid:4449060,user:Khalil Laleh,version:2,visible:true,tag:[2]}
Pipeline worker done without errors:: true
all items extracted
END ETL PROCESSOR
+ extracted 47 entries (0 entries/sec) - 47 entries -> loaded 47 bytes (0 bytes/sec) Total time: 65ms [0 warnings, 0 errors]

what can i do ?

@lvca
Copy link
Member

lvca commented Jan 27, 2017

@saeedtabrizi can you see where the NPE originated from?

@saeedtabrizi
Copy link
Contributor

saeedtabrizi commented Jan 27, 2017 •
edited
Loading

Hi @lvca as i explained in #7107 (comment) , after some tries and spending some hours , now i can to transfer my data from .osm xml files to orientdb but i realized the documentaion of ETL is not suitable and true .
I recommend review the documentation of ETL again .

but in the title of this issue , i found if you dont use orientdb as loader or using output as a loader you got this this exception as i explaind .

Thanks
Saeed

@robfrank
Copy link
Contributor

Hi, I replicated the problem. The workaround is to avoid "dbAutoDropIfExists": true,

I tested with previous versions and the same problem is there since a long time. I'll fix it for next release.

robfrank added a commit that referenced this issue Jan 30, 2017
@robfrank
fixes auto drop and creation in case of graph database
f3fd829 
refs #7109
@robfrank
Copy link
Contributor

Just pushed the fix, you can download the etl module 2.2.16-SNAPSHOT and drop it inside the lib folder. Delete the 2.2.15 jar before!

https://oss.sonatype.org/content/repositories/snapshots/com/orientechnologies/orientdb-etl/2.2.16-SNAPSHOT/

Please close the issue if the fix solves your problem

@robfrank robfrank added this to the 2.2.x (next hotfix) milestone Jan 30, 2017
@robfrank robfrank mentioned this issue Jan 30, 2017
Closed
4 tasks
@saeedtabrizi
Copy link
Contributor

Many Thanks @robfrank . i will check it .

@robfrank robfrank modified the milestones: 2.2.x (next hotfix), 2.2.16 Feb 2, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@robfrank robfrank

Labels
bug waiting reply
Milestone
2.2.16
Development

No branches or pull requests
5 participants
@lvca @robfrank @saeedtabrizi @prjhub @ahr12