Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add valuesort, new control result mode and value normalizer #237

Merged
merged 9 commits into from
Dec 20, 2024

Conversation

Omega359
Copy link
Contributor

I've adding some functionality to support running the sqlite test suite against another database and this PR is the initial part of that work. This PR adds:

  • valuesort from PR test against sqlite's test suite #123
  • a new control resultmode of either valuewise or rowwise (defaulting to the latter)
  • a new value normalizer to allow for different value normalization for different files

The last point is a breaking API change so I took the liberty to rev the version #. If that isn't desired feel free to revert that portion of this PR.

With this PR I am able to run a slightly cleansed sqlite test suite using the Datafusion sqllogictest harness.

# Conflicts:
#	Cargo.lock
#	Cargo.toml
@Omega359
Copy link
Contributor Author

@xxchan - any chance this could get a review?

@alamb
Copy link
Contributor

alamb commented Dec 20, 2024

For context, we are trying to run DataFusion against the entirety of the sqlite test corpus:

Copy link
Member

@skyzh skyzh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks, and happy holiday :)

A general question: does sqlite only do string comparison? What's their behavior of valuesort on an integer column? Currently we only sort by string, and could order 10 before 2. If they handle column types other than string, we probably need to add a custom comparator in the future?

query III rowsort
select * from example_sort
----
1 10 2333
10 100 2333
2 20 2333
2 20 2333
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you please also help add a test case for valuesort? (might require some modifications for FakeDB)

@Omega359
Copy link
Contributor Author

Omega359 commented Dec 20, 2024

LGTM, thanks, and happy holiday :)

Thanks, and you too!

A general question: does sqlite only do string comparison? What's their behavior of valuesort on an integer column? Currently we only sort by string, and could order 10 before 2. If they handle column types other than string, we probably need to add a custom comparator in the future?

From their docs:

Sort comparisons use strcmp() on the rendered ASCII text representation of the values. Hence, "9"
sorts after "10", not before. The "valuesort" mode works like rowsort except that it does not honor 
row groupings. Each individual result value is sorted on its own.

So it's the same string comparison as what is done here.

@skyzh skyzh merged commit ac188cb into risinglightdb:main Dec 20, 2024
4 checks passed
@skyzh
Copy link
Member

skyzh commented Dec 20, 2024

just released a new version :)

@alamb
Copy link
Contributor

alamb commented Dec 21, 2024

just released a new version :)

Wow -- talk about speedy service with a smile. Thank you @skyzh -- very much apprecaited

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants