All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
- runner: Added a
Normalizer
type for normalizing result values. A new functionwith_normalizer(normalizer: Normalizer)
has been added to the Runner to allow for specifying a custom Normalizer. The existing default normalizer is available via therunner::default_normalizer(..)
function. - parser: Added a new control mode
resultmode
that controls whether the results are invaluewise
orcolumnwise
mode. The default iscolumnwise
which means results are in columns.valuewise
means the results are in a single column (sqlite test style). - parser: Added
valuesort
sort mode. Thevaluesort
mode works like rowsort except that it does not honor row groupings. Each individual result value is sorted on its own.
Breaking change:
- The
Validator
type used in various function in Runner implementation has a new required fieldNormalizer
that is used to normalize result values.
- feat(bin): add opt
--keep-db-on-failure
- Refine the behavior of
update_record_with_output
/--override
- runner: Previously,
query
returning 0 rows will becomestatement ok
. Now it returnsstatement count 0
. - bin: Now
--override
will not change the type chars ofquery <types>
, since in practice it becomes?
s which might cause confusion.
- runner: Previously,
- runner:
statement count <n>
is incorrectly handled when the result is aquery
.
- engines/bin: fix compatibility with the new tokio-postgres minor version.
- engines/bin: support MySQL engine
Breaking changes:
- runner:
RecordOutput
is now returned byRunner::run
(orRunner::run_async
). This allows users to access the output of each record, or check whether the record is skipped. - runner(substitution): add a special variable
__NOW__
which will be replaced with the current Unix timestamp in nanoseconds. - runner(substitution): for
system
commands, we do not substitute environment variables any more, because the shell can do that. It's necessary to escape like\\
any more.$__TEST_DIR__
, and are still supported. - runner(system): change
sh
tobash
.
- runner: add logs for
system
command (with targetsqllogictest::system_command
) for ease of debugging.
- fix(runner): when running in parallel, the runner will correctly inherit configuration like
sort_mode
andlabels
from the main runner.
- bump dependencies
- feat(bin): hide
INFO
level log by default
- fix(bin):
halt
is not handled.
- bin: When using
-j <jobs>
to run tests in parallel, add a random suffix to the temporary databases. This is useful if the test is manually canceled, but you want to rerun it freshly. Note that if the test failed, the database will be dropped. This is existing behavior and unchanged. - bin: replace
env_logger
withtracing-subscriber
. You will be able to see the record being executed withRUST_LOG=debug sqllogictest ...
. - runner: fix the behavior of background
system
commands (end with&
). In0.20.0
, it will block until the process exits. Now we return immediately.system ok sleep 5 &
-
Show stdout, stderr when
system
command fails. -
Support matching stdout for
system
system ok echo "Hello, world!" ---- Hello, world!
Currently, only exact match is supported. Besides, the output cannot contain more than one blank lines in between. The record ends with two consecutive blank lines.
Some minor Breaking changes:
- Add field
stdout
toparser::Record::System
andrunner::RecordOutput::System
, and mark them as#[non_exhaustive]
. - Change trait method
AsyncDB::run_command
's return type fromstd::process::ExitStatus
tostd::process::Output
.
- Add field
- parser:
include
now returns error if no file is matched.
- parser: refactor
expect
field in sqllogictest parser to make it easier to work with.
-
Support matching multiline error message under
----
for bothstatement error
andquery error
.query error SELECT 1/0; ---- db error: ERROR: Failed to execute query Caused by these errors: 1: Failed to evaluate expression: 1/0 2: Division by zero
The output error message must be the exact match of the expected one to pass the test, except for the leading and trailing whitespaces. Users may use
--override
to let the runner update the test files with the actual output.Empty lines are allowed in the expected error message. As a result, the message must end with two consecutive empty lines.
Breaking changes in the parser:
- Add new variants to
ParseErrorKind
. Mark it as#[non_exhaustive]
. - Change the type of
expected_error
fromRegex
toExpectedError
, which is either a inlineRegex
or multilineString
.
- Add new variants to
- fix(runner): fix parallel testing db name duplication. Now we use full file path instead of filename as the temporary
db name in
run_parallel_async
.
- bin: support envvars
SLT_HOST/PORT/DB/USER/PASSWORD
-
Support environment variables substitution for SQL and system commands. For compatibility, this feature is by default disabled, and can be enabled by adding
control substitution on
to the test file.control substitution on query TTTT SELECT '$foo' -- short , '${foo}' -- long , '${bar:default}' -- default value , '${bar:$foo-default}' -- recursive default value FROM baz; ---- ...
Besides, there's a special variable
$__TEST_DIR__
which is the path to a temporary directory specific to the current test case. This can be helpful if you need to manipulate some external resources during the test.control substitution on statement ok COPY (SELECT * FROM foo) TO '$__TEST_DIR__/foo.txt'; system ok echo "foo" > "$__TEST_DIR__/foo.txt"
Changes:
- (parser) Breaking change: Add
Control::Substitution
. MarkControl
as#[non_exhaustive]
. - (runner) Breaking change: Remove
enable_testdir
. For migration, one should now enable general substitution by thecontrol
statement and use a dollar-prefixed$__TEST_DIR__
.
- (parser) Breaking change: Add
-
Support running external system commands with the syntax below. This is useful for manipulating some external resources during the test.
system ok echo "Hello, world!"
The runner will check the exit code of the command, and the output will be ignored. Currently, only
ok
is supported.Changes:
- (parser) Breaking change: Add
Record::System
, and correspondingTestErrorKind
andRecordOutput
. MarkTestErrorKind
andRecordOutput
as#[non_exhaustive]
. - (runner) Add
run_command
toAsyncDB
trait. The default implementation will run the command withstd::process::Command::status
. Implementors can override this method to utilize an asynchronous runtime such astokio
.
- (parser) Breaking change: Add
-
fix(runner): fix database name duplication for parallel tests by using the full path of the test file (instead of the file name) as the database name.
- fix(bin): fix error context display. To avoid stack backtrace being printed, unset
RUST_BACKTRACE
environment variable, or use pre-built binaries built with stable toolchain instead.
- fix(bin): do not print stack backtrace on error
- fix
statement error
unexpectedly passed when result is a successfulquery
. Similarly for expectedquery error
but successfulstatement ok
.
- Allow multiple connections to the database in a single test case, which is useful for testing the transaction
behavior. This can be achieved by attaching a
connection foo
record before the query or statement.- (parser) Add
Record::Connection
. - (runner) Breaking change: Since the runner may establish multiple connections at runtime,
Runner::new
now takes aimpl MakeConnection
, which is usually a closure that returns a try-future of theAsyncDB
instance. - (bin) The connection to the database is now established lazily on the first query or statement.
- (parser) Add
- We enhanced how
skipif
andonlyif
works. Previously it checks againstDB::engine_name()
, andsqllogictest-bin
didn't implement it.- (parser) A minor breaking change: Change the field names of
Condition:: OnlyIf/SkipIf
. - (runner) Add
Runner::add_label
. Now multiple labels are supported (DB::engine_name()
is still included). The condition evaluates to true if any of the provided labels match theskipif/onlyif <label>
. - (bin) Add
--label
option to specify custom labels.
- (parser) A minor breaking change: Change the field names of
Runner::update_test_file
properly escapes regex special characters.
- Support postgres options.
sqllogictest-bin
now uses the strict validator to update records (the runner still doesn't check schema).- The query syntax now allows optional columns (
query\n
without any column arguments).
- customizable column types and validators
- support multiple files as input in cli
- remove unnecessary debug
- fix parsing for trailing comments
This release contains some minor fixes.
- fix: use
Vec<Vec<String>>
for external engine (JDBC) - fix: Use
lines()
instead ofsplit('\n')
inparse_inner
. So the behavior can be correct on Windows. - fix: parse DML with
returning
as a query - A minor breaking change:
update_test_file
takes&mut Runner
instead of its ownnership.
-
Improve the ability to unparse and update the test files. Mainly add
update_record_with_output
andupdate_test_file
to the library.More details:
- Add
impl Display
forRecord
(refactorunparse
). - Add
Record::Whitespace
so the whitespace in the original files can be reconstructed duringunparse
. - Add tests for unparsing and updating records.
- Refactor and fix the behavior about newlines and
halt
for CLI options--override
and--format
.
- Add
-
Fix:
hash-threshold
should be compared with the number of values instead of the number of rows. -
Breaking change: The type of
Validator
is changed fromfn(&Vec<String>, &Vec<String>) -> bool
tofn(&[Vec<String>], &[String]) -> bool
. Also added adefault_validator
.
Thanks to the contributions of @alamb and @xudong963 .
- Improve the format and color handling for errors.
- Support
hash-threshold
. - Fix
statement count <n>
for postgres engines. - Breaking change: use
Vec<Vec<String>>
instead ofString
as the query results byDB
. This allows the runner to verify the results more precisely.- For
rowsort
, runner will only sort actual results now, which means the result in the test cases should be sorted.
- For
- Breaking change:
Hook
is removed. - Breaking change:
Record
and parser's behavior are tweaked:- retain
Include
record when linking its content - keep parsing after
Halt
- move
Begin/EndInclude
toInjected
- retain
- Added CLI options
--override
and--format
, which can override the test files with the actual output of the database, or reformat the test files.
- Support checking error message using
statement error <regex>
andquery error <regex>
syntax.- Breaking change:
Record::Statement
,Record::Query
andTestErrorKind
are changed accordingly.
- Breaking change:
- Fix:
--external-engine-command-template
should not be required
- Add support for external driver.
- Support more type in postgres-extended.
- Record file stack in location.
- Use one session for each file in serial mode.
- Support registering hook function after each query.
- Support load balancing of multiple addr.
- Integrate with libtest-mimic. Add the macro
sqllogictest::harness!
. - Improve error handling for parser.
- Add parallel running to
Runner
. - Drop database after parallel run.
- Support postgres extended mode
- Separate sqllogictest runner to sqllogictest-bin
- Add timestamp to junit. (#57)
- Add
sleep
function toAsyncDB
. (#61) - Fix panic without junit. (#58)
- Remove unsupported characters from junit test name.
- Add junit support. Use
--junit <filename>
to generate junit xml.
- Fix expanded
include
wildcard file name. (#52)
- Support wildcard in
include
statement. (#49)
- Show diff instead of actual / expected data on failed. (#51)
- Print empty strings as "(empty)"
- Support parallel sqllogictest
- Panic if no test file is found
- New test UI for sqllogictest binary
- Support set console color on the test UI with
--color
parameter
- Async interface
AsyncDB
for SQL runners. - support evaluating
skipif
andonlyif
conditions - support file-level sort mode control syntax
- supports custom validator
- A command-line tool to run scripts from file against postgres-compatible databases.
- Support
sleep
andinclude
statement.
- Add file location to the error message.
- Runner returns error type instead of panic.
- Basic sqllogictest parser and runner.