Jepsen tests: current state

[zamazan4ik]

Hi!

I have found that Tarantool supports somehow Jepsen test suite (https://github.com/tarantool/jepsen.tarantool). Also, already there are some CI workers which do Jepsen testing. E.g. this one - https://github.com/tarantool/tarantool/actions/workflows/jepsen-single-instance.yml

For me as for the end user is not clear, what is the current state of Jepsen Testing for Tarantool. Passing Jepsen test suite successfully is kinda imporatant point for the deciding, is the database mature enough or not.

My suggestions are the following:

• put somewhere into the documentation current state of Jepsen testing for Tarantool
• describe all Jepsen workers. Now in CI I see 4 different Jepsen-related configurations. Some of them are broken and it is not clear - is it by design or due to some bugs in Tarantool itself. E.g. Tarantool Cluster is "red" and it could be a red flag for choosing Tarantool for cluster setup. Single instance is good but cluster configuration is more interesting from the Jepsen point of view.

Thanks!

[Totktonada]

AFAIK, we do some experiments around Jepsen testing, but it is more like a PoC. There is no active work in this direction at the moment.

[zamazan4ik]

@Totktonada so what kind of testing for Tarantool in different distributed cases is performed? Could you anyhow compare test suite to the Jepsen?

[ligurio]

@zamazan4ik Alexander, thanks for your interest to Tarantool! I'm an author of Jepsen tests, and I'm glad that you highlighted these tests.

In short, our efforts with Jepsen tests are on pause. We plan to continue using them in the future.

• put somewhere into the documentation current state of Jepsen testing for Tarantool

We implemented a number of tests: register, counter, bank and set with a number of nemeses, described in the README, see ¹. These tests and nemeses were ported from another testsuites developed by Kyle Kingsbury, author of Jepsen framework.

• describe all Jepsen workers. Now in CI I see 4 different Jepsen-related configurations. Some of them are broken and it is not clear - is it by design or due to some bugs in Tarantool itself. E.g. Tarantool Cluster is "red" and it could be a red flag for choosing Tarantool for cluster setup. Single instance is good but cluster configuration is more interesting from the Jepsen point of view.

Regression tests are a tools used by developers, not by users. It is not clear for me why do we need to describe tested configurations, usefulness of tests etc. and keep this information up to date. For users and our customers we have a special page where we described guarantees provided by Tarantool, see ². It is better to rely on information provided by documentation than looking into test results and try to intepret it's results. Jepsen tests are generative tests with active using fault injection. Believe me, it is not always trivial to interpret results right without being involved deeply.

BTW, could you share information about your scenarios used with Tarantool? What is a configuration of your cluster? What a level of transaction isolation is required in your applications?

Footnotes

1. https://github.com/tarantool/jepsen.tarantool#options ↩

2. https://www.tarantool.io/en/doc/latest/book/box/atomic/transaction_model/ ↩

[zamazan4ik]

BTW, could you share information about your scenarios used with Tarantool? What is a configuration of your cluster? What a level of transaction isolation is required in your applications?

We do not have yet a Tarantool installation, so I cannot provide you such information :) We are just evaluating different databases with a possibility in the future to bring Tarantool into our internal technology portfolio.

Why I am interested in Jepsen tests especially. You know, passing successfully Jepsen tests is a really good sign of maturity for any distributed systems. For me as for a person, who evaluates different technologies, such a mark is a great approximation that I can somehow trust to the distributed model of a database. That is the reason why I am asking about a Jepsen mention somewhere in the documentation.

You are completely right that for the users is not clear to check different CI configurations, etc - for us is much easier just read the documentation. However, after reading the documentation I always check list of open issues with some keywords like "crash", "consistency",etc. And if I find something related to Jepsen in a failing state (as Tarantool CI Jepsen-related workers) - it is at least a sign for me to investigate a little futher regarding bugs in a cluster mode.