-
Notifications
You must be signed in to change notification settings - Fork 211
Testing, Continuous Integration, and Deployment
We use autotest for executing a testsuite. The test suite is located in souffle/tests.
The testscript testsuite is produced by the makefile file invoking
make check
either in the main directory of souffle or in the tests directory.
The testsuite script either runs all tests, individual tests or can list tests. More information on how to run tests with testsuite can be found here and you obtain the help message with
souffle/tests/testsuite -h
For example, knowing the test-number for a given test that runs a test program with a given environment (compiler/interpreter/parallel) can be executed as follows:
souffle/tests/testsuite <testnumber>
A test with its key can also be executed with
tests/testsuite -k <testname>
If the test fails, you will find the logs in the director souffle/tests/testsuite.dir/<testnumber>.
In this directory, there will be a run script that you can use to re-run the test. It basically just calls souffle/tests/testsuite <testnumber>.
To obtain the list of tests with the configuration, use the command
souffle/test/testsuite --list
The command
souffle/tests/testsuite --recheck
reruns only the failed tests in the testsuite.
If you want to run test ranges you can invoke testsuite as follows:
souffle/tests/testsuite 100-112 240 256-259 -d -j8
where -j8 means eight tests in parallel, and leave the results in the souffle/testsuite.dir/
if they pass (flag -d).
You can run tets via make using
TESTSUITEFLAGS=-j8 SOUFFLE_CATEGORY=Syntactic,Semantic SOUFFLE_CONFS='-j1,-c -j1' make check -j8
where SOUFFLE_CATEGORY/SOUFFLE_CONFS is used when building tests/testsuite.
Note that running make check again with different environment variables does not rebuild testsuite - you need to first make clean. The environment variable TESTSUITEFLAGS is used to pass arguments to the script souffle/tests/testsuite.
TBD. The directory structure of tests. Positive/Negative tests. The directory structure of a single test/naming convention for input/output/error files.
We have C++ unit tests in souffle/src/tests.
TBD.
All pull requests must be reviewed and approved before merging with the main branch. The ideal pull request contains only relevant commits with useful subjects (Allow use of nested aggregates vs changes).
Pull requests are merged rather than squashed to preserve commit history and to ensure due credit to contributors so it is helpful if the commit subjects are informative. Pull requests are also not rebased as occasionally there are separate branches being merged, so the merge is informative, but more often this is to make clear the extents of the block of new code forming the pull request.
All pull requests are automatically tested using travis-ci. An initial code style test is performed using clang-format. Next, a set of tests equivalent to running make check locally, are run on Ubuntu 16.04 and OSX, with gcc and clang. These tests are further divided into categories to ensure completion within travis timeouts. The choice of OS and compiler version is governed mostly by the limitations of working with Travis CI. An additional test is run with Ubuntu 16.04 using 64bit RamDomains. A final set of tests checks that install (make install), deployment (make package), and documentation (make doxygen) work as intended.
All travis tests can be enabled in forked repositories.
The most frequent maintenance issues with these tests is restarting to handle spurious errors, such as transient setup problems, and updating outdated dependencies.
A more comprehensive suite of tests is run on soufflé-lang servers covering a wider range of OS and compiler combinations. These tests also run clang-tidy as a linter, and run a code coverage analysis.
The Pull Request Builder plugin is used to run tests on every PR.
- 'make check' on Centos7 with gcc7 installed
- 'make check' on Ubuntu LTS
- 'make check' on the latest Ubuntu development release.
- 'make check' with '--enable-sanitise-memory' to test for new memory leaks
- clang-tidy to show common code issues (uninitialised member variables, etc)
- 'make distcheck' followed by make install then running a small test dl (Note that distcheck runs make check with no parallel options so this build is slower.)
- 'make check' with coverage enabled, the results of which are uploaded to Codecov
Transient failures, such as due to network problems, can be worked around by rerunning the tests (an admin can comment on the PR with a message containing 'test this please') or by adding new commits to the PR.
Once a PR is successfully merged, a similar set of tests is run with more comprehensive options (OS and compiler) to attempt to identify rarer, or transient, errors. These tests can be manually rerun by admins. The post-merge tests also include coverage, to provide a baseline coverage, and benchmarks to track performance (both memory and CPU use).
Code coverage results from the Jenkins tests are passed to Codecov for further evaluation.
When a release is made, packages will be built and added to the GitHub Releases page.
Builds from a successfully merged pull request will be added to the souffle-lang repositories on PackageIO.