Fixing a CI testing strategy for the adapter

[IshaanDesai]

Description

Currently the adapter is tested for the last two DuMux releases and the last three preCICE releases:

dumux-adapter/.github/workflows/build-and-test.yml

Lines 13 to 15 in 1159c0c

	matrix:
	dumux_version: [3.4, 3.5]
	precice_version: [2.3.0, 2.4.0, 2.5.0]

Additionally the master branch of DuMux (equivalent of a development branch) is tested against the last two releases of preCICE:

dumux-adapter/.github/workflows/build-and-test-dumux-master.yml

Lines 13 to 15 in 1159c0c

	matrix:
	precice_version: [2.4.0, 2.5.0]
	dune_version: [2.8]

These combinations were decided after a brief discussion with @ajaust, but can now be revisited based on the release strategy discussion in #23, as the examples are routinely updated and no longer work with older DuMux versions. How many versions should we test for the adapter?

[ajaust]

Just to mess things up more, you could also test against several DUNE versions, but I am not sure if that is worth it. ;)

Tests against DuMuX and preCICE

Two questions to start with:

• What are typical approaches for other adapters?
• What versions does DuMuX test and is their testing strategy that may make sense for this adapter?

I guess for preCICE versions, testing with the most recent version is enough. If there is a major release (preCICE v2 to v3) it would make sense to test with the two most recent major releases during the transitioning phase. The adapter is not using "fancy" features, as far as I remember, that have come with any recent preCICE minor release so I would assume that testing with the most recent preCICE version would imply compatibility with preCICE 2.0.0.

Regarding DuMuX: Maybe the DuMuX developers have an idea how often their users are updating DuMuX? People on the DuMuX mailing list seem to use (very) old DuMuX versions quite regularly.

I am not sure if it would make sense to test with two (or more) DUNE version. Maybe when there is a new release and people transition.

Tests against DuMuX master

I had added the the tests against DuMuX master as "canary" test. The idea was to be aware early that DuMuX brings some breaking changes and which should make it easier to quickly support new major versions. However, I am not sure how often I actually used this. If one keeps this test, I would assume testing against only one preCICE version (most recent?) should be enough.

[mathiskelm]

For DuMux we test the master/main/develop branch (against the latest release as well as master branch of DUNE) and the last (now last 2) releases of DuMux (against their respective DUNE releases).

Using different DUNE versions in the adapter testing is complicated but perhaps necessary. DuMux releases are explicitly compatible with only one DUNE release. E.g. with 3.7 we will be switching from DUNE 2.8 to 2.9. While 2.9 is backwards compatible with 2.8 and thus DuMux 3.6, the new DuMux release will use features not available in DUNE 2.8. DuMux master has already lost compatibility with DUNE 2.8. As long as we're not testing more than 2 adjacent DuMux releases, we could test both with the newer version of DUNE that they're both compatible with and then test against DuMux master with whatever release it requires?

While there will always be users of older versions of DuMux and the minor releases are not as compatible as for preCICE, the adapter itself should not be breaking very often. As you pointed out it will probably be the tests/examples that need to be adapted instead.

[uekerman]

What are typical approaches for other adapters?

Many open problems there still.

I would suggest to keep it simple: Test against exactly those versions, where you promise compatibility, see #23

Testing against development versions can be nice to get early feedback, but might not be sooo important. When there is a new DuMuX version, I would probably first test manually for compatibility and then see whether adjustments are necessary.

The main purpose of the tests are to have a safeguard when changing sth in the adapter: "I am a breaking any of my compatibility promises with this PR?"