Multi-Threading and Transactions in OrientDB

[bujoralexandru]

Hello, OrientDB community!

I won't bore you with formalities, so I will jump right in:

I have the following use-case:

I need to import a metric ton of XML-modeled data into the OrientDB Graph Database. This data is consisting of the following:

• simple data (that translates to plain vertices inside the graph, the key aspect of this data is that each entry, does not depend on the previous one, so this is a perfect example of optimizing via multi-threading [eg. having N threads export a chunk of TotalSize divided by N, on N transactions, each transaction NOT TOUCHING other transaction's vertices])
• and complex data (which translates to chunks of information, each chunk correlating the vertices imported from the simple data [eg. they add a lot of edges between the simple data]). Important is that, each such chunk is independent of the other, meaning, that it does not rely on the information provided by none other; this aspect told me that, besides the shared simple data, that is 100% only vertices, these chunks are disjunct, in the sense that the do not have conflicts like two chunks alter the same property of the same vertex.

Also, I use Ferma as an OGM and doing all my graph inquires via the Gremlin API of the OrientDB.

So, my answer seemed to me to be pretty straight forward: multi-threading. But it wasn't as such...

My problem:

I tried a first draft of implementing a single threaded export of the simple data and a multi-threaded export of the complex data.
Each chunk, that is disjunct, has it's own thread and it's own transaction,, but somehow I still get ConcurrentModificationException and several other cryptic exceptions from the OrientDB driver, stating that it cannot find vertices with ids #-1:3, #-1:12 and so on, exceptions that I couldn't make anything of.

My question:
Can I implement what I want using multi-threading? If yes, what am I doing wrong with my initial approach? If no, what are my implementation alternatives to cut the time of my import as much as possible?

PS:
I read the 1st 4 pages of Google regarding OrientDB and multi-threading. I really would appreciate an experimented user's own opinion, not a let me google it for you type of response. Also, if there is any developer that managed to implement some multi-threaded operations using OrientDB transactions, I would be so thankful if he or she can share that code with me.

Thank you so much for reading through my post and I hope this discussion may reach other developers that are troubled by a similar problem.

Cheers,
Alex

[luigidellaquila]

Hi Alex,

Welcome to OrientDB Community!

Multi-threading is not a complex topic as long as you stick to a couple of basic rules.

The first one is: never ever share connections and records between threads, always open, use and close a session in the same thread; if you need to share records between threads, pass RIDs and reload the record from inside the other thread.

This is enough to be safe with the API contracts.

The second rule is to consider the strengths and the limits of OrientDB transaction model. Optimistic transactions trade locking (ie. server overhead, less parallelism) with failures (ie. the need to retry transactions in case of collisions).
In OrientDB, each record has a version (a number that starts from zero and increments at each update); a transaction succeeds only if all the records involved have the "right" version number, ie. the same version number present in the storage. If two transactions happen in parallel, one will fail for with OConcurrentModificationException, so you you have to replay it from scratch.

The best way to have a good multi-threading with OrientDB is to keep the transaction scopes isolated, eg. if you have updates on two different schema classes, you can have one thread per class and you will never have collisions.

Another thing: TinkerPop adds overhead and complexity to the whole thing, I'd strongly suggest to use the multi-model API directly (it has the exact same concepts and it's not so different) https://orientdb.com/docs/3.0.x/java/Java-MultiModel-API.html

Thanks

Luigi